Text information compression method and compression device for index distributed database

Abstract

The invention discloses a text information compression method and compression device for an index distributed database, and the method comprises the steps: obtaining a to-be-processed text, carrying out the preprocessing of the to-be-processed text, and giving a plain text in a preset format; performing multi-stage semantic paragraph division on the plain text in the predetermined format, and giving a corresponding semantic paragraph set; performing fragment division and mapping processing considering weight and position on each semantic paragraph in the semantic paragraph set, and giving a signature identifier of each semantic paragraph; and indexing the signature identifier, and generating compressed text information for indexing the distributed database in combination with the signature identifier and the identifier information of the semantic paragraph. Misjudgment caused by chapter structure change can be avoided; and unbiased estimation of similarity is kept while high compression is performed, so that text subjected to semantic rewriting, partial plagiarism or structure recombination can be recognized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of text processing and information retrieval, and particularly relates to a text information compression method and compression device for indexing a distributed database. BACKGROUND

[0002] With the rapid development of digital technology, text data is growing explosively. In the fields of academic research, enterprise office, and Internet content dissemination, problems such as text plagiarism and repeated content are increasingly prominent. Text duplication detection technology has become a key support for ensuring information originality and compliance.

[0003] Currently, there are various methods in the field of text duplication detection. For example, the publication number CN115759082A provides a text duplication detection method and device based on an improved Simhash algorithm. The Simhash algorithm is used to calculate the similarity of texts. However, the algorithm has poor performance when facing problems such as word replacement, sentence transformation, or rewriting without changing the original meaning. The weight calculation of the word segmentation vector aggregation is usually set to 1 or the number of word segmentation occurrences, which can easily cause information loss, resulting in reduced accuracy of the final text vector. Moreover, the algorithm cannot well represent the word segmentation distribution information and context semantic information. For texts containing the same word segmentation but different meanings, the text vectors generated after adjusting the order of word segmentation are the same, which can easily result in a large number of overlapping text vectors. In addition, the publication number CN118885602A provides a text duplication detection method and system based on the LLM2Vec algorithm. The algorithm uses string matching-based algorithms such as the longest common subsequence algorithm, which is easily disturbed by text rewriting and synonym replacement. The tf-idf-based method cannot capture deep semantic information of texts, resulting in insufficient semantic understanding of text content and limited duplication detection accuracy.

[0004] Therefore, there is an urgent need for a text compression / duplication detection solution that can deeply integrate semantic understanding, efficiently retrieve, and have continuous optimization capability, aiming to solve the above technical defects and achieve efficient, accurate, and dynamically optimized text duplication detection. SUMMARY

[0005] In view of the defects in the above-mentioned prior art, the present application provides a text information compression method and compression device for indexing a distributed database. The method can avoid misjudgment caused by changes in the structure of the text. The method can perform high compression while maintaining unbiased estimation of similarity, which can help identify texts that have been semantically rewritten, partially plagiarized, or structurally reorganized.

[0006] In a first aspect, the present application provides a text information compression method for indexing a distributed database, comprising: obtaining a text to be processed and performing preprocessing to obtain a pure text in a predetermined format; perform multi-level semantic paragraph division on the pure text in the predetermined format, and give the corresponding semantic paragraph set; perform fragment division and mapping processing of each semantic paragraph in the semantic paragraph set considering weight and position, and give the signature identification of each semantic paragraph; perform indexing processing on the signature identification, and generate compressed text information for indexing the distributed database in combination with the signature identification and the identification information of the semantic paragraph.

[0007] Further, the text to be processed is obtained and preprocessed to give the pure text in the predetermined format, including: obtain multi-source text to be processed; perform unified coding on the multi-source text to be processed to generate an initial text; perform cleaning and denoising processing on the initial text to obtain the pure text in the predetermined format.

[0008] Further, the pure text in the predetermined format is subjected to multi-level semantic paragraph division to give the corresponding semantic paragraph set, including: perform punctuation and blank line recognition on the pure text in the predetermined format to determine the positions and types of all punctuation in the pure text, and the positions of blank lines formed by two or more consecutive line breaks; determine the forced segmentation points based on the positions and types of all punctuation in the pure text, and the positions of blank lines formed by two or more consecutive line breaks; perform one-time text cutting on the pure text in the predetermined format according to the forced segmentation points to obtain a plurality of basic text segments; perform processing and analysis on the basic text segments through a sliding window to give a cosine similarity matrix; determine the cosine similarity change rates of a plurality of continuous sliding windows based on the cosine similarity matrix, and give the semantic break points in combination with a preset amplitude threshold; perform two-time text cutting on the plurality of basic text segments based on the semantic break points to give an initial semantic paragraph set; filter the initial semantic paragraph set based on a preset filtering strategy to give a final semantic paragraph set.

[0009] Further, each semantic paragraph in the semantic paragraph set is subjected to fragment division and mapping processing considering weight and position to give the signature identification of each semantic paragraph, including: perform word segmentation processing on each semantic paragraph in the semantic paragraph set to generate a word sequence of each semantic paragraph; perform similarity calculation on each word sequence, and combine adjacent words with a similarity reaching a predetermined threshold to give a first semantic fragment set; perform fragment division on each semantic paragraph in the semantic paragraph set based on a sliding window determination condition to give a second semantic fragment set; merge and filter the first semantic segment set and the second semantic segment set to obtain a final semantic segment set; Based on the preset hash function family, identify and multiple fusion weight processing is performed on each semantic segment in the final semantic segment set, and the signature identification of each semantic paragraph is given.

[0010] Further, similarity calculation is performed on each word sequence, and adjacent words with a similarity reaching a predetermined threshold are merged to give a first semantic segment set, including: Each word sequence is vectorized by a pre-trained language model to give a word vector for each word in the word sequence; The word vectors of adjacent words in each word sequence are calculated to obtain the similarity of adjacent words; Adjacent words with a similarity exceeding a predetermined threshold are combined into a merged word; The merged word and the word that has not been merged are respectively taken as a semantic segment and integrated to obtain the first semantic segment set.

[0011] Further, based on the sliding window determination condition, each semantic paragraph in the semantic paragraph set is segmented to give a second semantic segment set, including: Determine the character sequence of each semantic paragraph; Based on all character sequences, the occurrence probability of each character in each character sequence is given; Based on the characters possessed by each character sequence and in combination with the corresponding occurrence probability, the information entropy of each semantic paragraph is given; Based on the information entropy of each semantic paragraph, the corresponding sliding window size is determined; Based on the corresponding sliding window size, each semantic paragraph is segmented to obtain the second semantic segment set.

[0012] Further, based on the preset hash function family, the semantic segments of each semantic paragraph are identified and multiple fusion weight processing is performed to give the signature identification of each semantic paragraph, including: Each semantic segment of each semantic paragraph is calculated and fused by multiple weight calculation methods to give the fusion weight of each semantic segment; Each semantic segment is spliced with its position information to give a position-enhanced semantic segment; Based on each hash function in the preset hash function family using different seed values, hash calculation is performed on all position-enhanced semantic segments in each semantic paragraph to give the hash value of each position-enhanced semantic segment corresponding to different hash functions; Based on the seed value of each hash function, the identification of the text to be processed, and the fusion weight of each semantic segment, an adjustment value of each position-enhanced semantic segment corresponding to different hash functions is given; The hash values of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph are fused with the corresponding adjustment values respectively to give the weighted hash values of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph. The minimum weighted hash value of each hash function is selected from all the weighted hash values of the semantic paragraph to give the signature identification of each semantic paragraph.

[0013] Further, the signature identification is indexed, and combined with the signature identification and the identification information of the semantic paragraph, a compressed text information for indexing the distributed database is generated, including: Based on a preset division strategy, the signature identification is divided into multiple wave bands; The corresponding byte sequence of each wave band is calculated to give the bucket key corresponding to each wave band; The signature identification, the identification information of the signature identification corresponding to the text to be processed, and the identification information of the signature identification corresponding to the semantic paragraph are stored in the index distributed database by taking the bucket key as an index.

[0014] Further, the method further comprises: The text to be checked for duplication is sequentially preprocessed, multi-level semantic paragraph division, segment division, mapping processing and indexing processing to give the signature identification and the indexing processing result of the text to be checked for duplication; Based on the indexing processing result of the text to be checked for duplication, the corresponding signature identification is queried from the index distributed database; Based on the signature identification of the text to be checked for duplication and the signature identification of the index distributed database, a text duplication checking result is given.

[0015] In a second aspect, the present application also provides a text information compression device for indexing a distributed database, which adopts the above-mentioned text information compression method, and the device comprises: A preprocessing module is used to obtain a text to be processed and pre-process it to give a pure text in a predetermined format; A paragraph division module is used to divide the pure text in the predetermined format into multi-level semantic paragraphs to give a corresponding semantic paragraph set; An identification generation module is used to divide and map each semantic paragraph in the semantic paragraph set considering the weight and position to give a signature identification of each semantic paragraph; An indexing and storage module is used to index the signature identification, and generate a compressed text information for indexing the distributed database in combination with the signature identification and the identification information of the semantic paragraph.

[0016] Further, the preprocessing module is configured to: obtain a plurality of source texts to be processed; perform uniform coding on the plurality of source texts to be processed to generate an initial text; perform cleaning and denoising on the initial text to obtain a pure text in a predetermined format.

[0017] Further, the paragraph division module is configured to: perform punctuation and line space recognition on the pure text in the predetermined format to determine the positions and types of all punctuation marks in the pure text and the positions of line spaces formed by two or more consecutive line spaces; determine forced segmentation points based on the positions and types of all punctuation marks in the pure text and the positions of line spaces formed by two or more consecutive line spaces; perform a first text cutting on the pure text in the predetermined format according to the forced segmentation points to obtain a plurality of basic text segments; perform processing and analysis on the basic text segments by using a sliding window to obtain a cosine similarity matrix; determine the cosine similarity change rates of a plurality of consecutive sliding windows based on the cosine similarity matrix, and give semantic break points in combination with a preset amplitude threshold; perform a second text cutting on the plurality of basic text segments based on the semantic break points to give an initial semantic paragraph set; filter the initial semantic paragraph set based on a preset filtering strategy to give a final semantic paragraph set.

[0018] Further, the identification generation module is configured to: perform word segmentation processing on each semantic paragraph in the semantic paragraph set to generate a word sequence of each semantic paragraph; perform similarity calculation on each word sequence, and combine adjacent words with a similarity reaching a predetermined threshold to give a first semantic segment set; perform segment division on each semantic paragraph in the semantic paragraph set based on a sliding window determination condition to give a second semantic segment set; perform mixed filtering on the first semantic segment set and the second semantic segment set to obtain a final semantic segment set; perform identification and multiple fusion weight processing on each semantic segment in the final semantic segment set based on a preset hash function family to give a signature identification of each semantic paragraph.

[0019] Further, the identification generation module is configured to: perform vectorization on each word sequence by using a pre-trained language model to give a word vector of each word in the word sequence; perform similarity calculation on the word vectors of adjacent words in each word sequence to obtain the similarity of adjacent words; combining adjacent words with similarity exceeding a predetermined threshold into a merged word; integrating the merged word and the word not subjected to merging as a semantic segment respectively to obtain a first set of semantic segments.

[0020] Further, the identification generating module is configured to: determine a character sequence of each semantic paragraph; based on all the character sequences, give an occurrence probability of each character in each character sequence respectively; based on the characters possessed by each character sequence and in combination with the corresponding occurrence probability, give an information entropy of each semantic paragraph; based on the information entropy of each semantic paragraph, determine a corresponding sliding window size; based on the corresponding sliding window size, perform segment division on each semantic paragraph to obtain a second set of semantic segments.

[0021] Further, the identification generating module is further configured to: perform weight calculation and fusion on the semantic segments of each semantic paragraph respectively through multiple weight calculation methods to give a fusion weight of each semantic segment; splice each semantic segment with its position information to give a position-enhanced semantic segment; based on each hash function in the preset hash function family adopting different seed values, perform hash calculation on all the position-enhanced semantic segments in each semantic paragraph respectively to give a hash value of each position-enhanced semantic segment corresponding to different hash functions; based on the seed value of each hash function, the identification of the text to be processed and the fusion weight of each semantic segment, give an adjustment value of each position-enhanced semantic segment corresponding to different hash functions; fuse the hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions with the corresponding adjustment value respectively to give a weighted hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions; select the minimum weighted hash value of each hash function from all the weighted hash values of the semantic paragraph to give a signature identification of each semantic paragraph.

[0022] Further, the index and storage module is configured to: based on a preset division strategy, divide the signature identification into multiple wave bands; perform calculation on the corresponding byte sequence of all the wave bands respectively to give a bucket key corresponding to each wave band; take the bucket key as an index to store the signature identification, the identification information of the text to be processed corresponding to the signature identification and the identification information of the semantic paragraph corresponding to the signature identification into an index distributed database.

[0023] The application provides a text information compression method and compression device for indexing a distributed database. (1) By semantic segmentation and generating high-distinguishable signature identification, accurate identification of semantic rewriting and partial plagiarism is realized. By dividing the text into paragraph units with complete semantics, misjudgment caused by changes in the chapter structure can be effectively avoided. And by the generated signature identification, the paragraph content can be highly compressed and the bias-free estimation of similarity is maintained, which helps to identify texts that have been rewritten, partially plagiarized or structurally reorganized.

[0024] (2) When generating the signature identification, the paragraph-level processing after semantic segmentation can accurately capture local similarity, further improving the identification of texts that have been rewritten, partially plagiarized or structurally reorganized.

[0025] (3) By the fragment division method of semantic unit merging and dynamic K-gram combination, the semantic integrity and local features of semantic fragments are considered; by position enhancement, the text structure information is retained, which helps to restore the paragraph structure and sentence order of the text, and is suitable for downstream tasks such as document layout restoration, multi-document alignment and text duplication detection that require structure information; by the weight setting method of fusing semantic weight, TF-IDF weight and part-of-speech weight, high weight is assigned to the core content, which is preferentially retained during compression and removes redundant information, which can better reflect the essential features of the text than the conventional compression method; the efficient signature generation method of weighted MinHash compresses the text into fixed-length signatures that retain key semantics and structure, which is more suitable for large-scale text compression. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of a text information compression method for indexing a distributed database is provided for the application; Figure 2 A flowchart of a semantic paragraph set is provided for an embodiment of the application; Figure 3 A flowchart of a signature identification is provided for an embodiment of the application; Figure 4 A schematic diagram of determining the final semantic fragment set is provided for an embodiment of the application; Figure 5 A flowchart of identification and multiple fusion weight processing is provided for an embodiment of the application; Figure 6 A schematic diagram of determining the signature component is provided for an embodiment of the application; Figure 7 A flowchart of generating compressed text information is provided for an embodiment of the application; Figure 8 A flowchart of text information duplication detection is provided for an embodiment of the application; Figure 9 A schematic diagram of a text information compression device for indexing a distributed database is provided. DETAILED DESCRIPTION

[0027] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings and specific embodiments of the specification. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two.

[0029] It should also be noted that the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the goods or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such goods or devices. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the goods or devices including the element.

[0030] As Figure 1 shown, the present application provides a text information compression method for indexing a distributed database, which can include: Obtaining the text to be processed and preprocessing to give the pure text of predetermined format; Carrying out multi-level semantic paragraph division on the pure text of predetermined format to give the corresponding semantic paragraph set; Carrying out fragment division and mapping processing considering weight and position on each semantic paragraph in the semantic paragraph set to give the signature identification of each semantic paragraph; Indexing the signature identification and combining the signature identification and the identification information of the semantic paragraph to generate the compressed text information for indexing the distributed database.

[0031] Wherein, when obtaining the text to be processed and preprocessing to give the pure text of predetermined format, it can include: Obtaining multi-source text to be processed; Uniformly encoding the multi-source text to be processed to generate the initial text; The initial text is cleaned and denoised to obtain a predetermined format of pure text.

[0032] In an actual application scenario, when preprocessing the text to be processed, the original text to be processed in different formats can be converted into unified UTF-8 encoded pure text, and irrelevant noise can be removed, so as to support input of texts / documents in multiple formats such as PDF, Word, HTML, TXT and Markdown. For PDF and Word format documents, Apache Tika can be used for parsing and extraction, and finally pure text can be obtained; HTML format documents can be parsed by Jsoup to extract valid text within the tag to obtain pure text; TXT and Markdown format documents directly read the original text. When cleaning and denoising, fixed text segments that appear more than a set number of times (such as headers and footers) can be identified, and noise can be removed in combination with the location characteristics of the text on the page. Based on a multilingual stop word dictionary and regular expressions, semantically meaningless words and special symbols can be filtered to ensure that the text content is pure and semantically complete for subsequent processing.

[0033] In order to divide the preprocessed pure text into semantic paragraphs with complete semantics and filter invalid paragraphs without actual compression meaning, and to avoid short text segments, data-intensive segments, and templated content from interfering with subsequent fingerprint (signature identifier) generation and similarity calculation, a two-level processing strategy of primary segmentation + advanced segmentation is used for semantic segmentation processing to divide semantic paragraphs. Specifically, as shown in Figure 2 The multi-level semantic paragraph division of the predetermined format of pure text is performed, and the corresponding semantic paragraph set is given, which can include: Punctuation and blank line recognition is performed on the predetermined format of pure text to determine the location and type of all punctuation in the pure text and the location of blank lines formed by two or more consecutive line breaks; Based on the location and type of all punctuation in the pure text and the location of blank lines formed by two or more consecutive line breaks, the forced segmentation points are determined. According to the forced segmentation points, the predetermined format of pure text is cut once to obtain a plurality of basic text segments. The plurality of basic text segments are processed and analyzed by a sliding window to give a cosine similarity matrix, including: slidingly selecting sentences for each basic text segment by a sliding window; each sentence is vectorized to give a corresponding semantic vector; the semantic vectors are processed based on the cosine similarity algorithm to give the semantic similarity between the sentences corresponding to adjacent sliding windows; the semantic similarity is integrated based on the sliding selection order of the sliding window to give the cosine similarity matrix. Based on the cosine similarity matrix, the cosine similarity change rate of a plurality of consecutive sliding windows is determined, and a semantic break point is given in combination with a preset amplitude threshold. perform secondary text cutting on the plurality of basic text segments based on semantic breakpoints, to give an initial semantic paragraph set; filter the initial semantic paragraph set based on a preset filtering strategy, to give a final semantic paragraph set. The preset filtering strategy includes: a length filtering strategy, a number proportion strategy, and a template matching strategy.

[0034] In actual application scenarios, the primary segmentation is a basic segmentation stage driven by a rule engine. The rule engine has two types of core rules built-in, i.e., punctuation segmentation rules and blank line detection rules. The punctuation segmentation rules cover the logic of forced segmentation points at the end of a sentence (such as a period, a question mark, and an exclamation mark). When a line break exists after the forced segmentation point at the end of a sentence, the text after the line break is divided into a new semantic paragraph. The punctuation segmentation rules also include the logic of segmentation hints within a sentence (such as a comma and a semicolon). The segmentation hint logic is only used as a potential segmentation reference and does not directly trigger segmentation. The blank line detection rules identify the text interval formed by two or more consecutive line breaks in pure text, and determine the interval position as a forced segmentation point. Through the combination of punctuation segmentation rules and blank line detection rules, the preprocessed pure text is preliminarily cut into a plurality of basic text segments.

[0035] The advanced segmentation uses the fine-tuned BERT-TextTiling model to perform semantic optimization on the basic text segments. During semantic optimization, each sentence in the window is converted into a semantic vector in units of sliding windows. Then, the cosine similarity algorithm is used to calculate the semantic similarity between sentences, and a cosine similarity matrix between sentences in the window is constructed. When the cosine similarity of three consecutive sliding windows decreases by more than a preset amplitude threshold (cosine similarity change rate), it is determined that the position is a semantic breakpoint, i.e., the semantic relevance of the preceding and following text is significantly reduced, and paragraph splitting is required. Based on these semantic breakpoints, the primary segmentation result is adjusted twice, i.e., the basic segments with coherent semantics are merged, and the segments with semantic breaks are split, to finally form an initial semantic paragraph set that is complete in semantics and coherent in logic.

[0036] After obtaining the initial semantic paragraph set, it is necessary to filter out invalid paragraphs in the initial semantic paragraph set. Specifically, a three-level filtering strategy (preset filtering strategy) is used to screen the initial semantic paragraph set, so that only valid paragraphs with actual value are retained.

[0037] The first-level filtering strategy is the length filtering strategy. There are short semantic paragraphs in the semantic paragraph set. Such short semantic paragraphs only contain scattered words or short sentences and cannot reflect complete semantics, which may lead to noise in subsequent signature identification generation and affect compression efficiency. Therefore, the length filtering strategy can be used for filtering and removal. Specifically, the total number of characters in each semantic paragraph is counted by character counting, and semantic paragraphs with a character number less than a preset threshold are removed. The second-level filtering strategy is a digital proportion filtering strategy. All digital characters in a semantic paragraph are extracted through a regular expression, a proportion of the digital characters in the total number of characters in the semantic paragraph is calculated, and when the proportion exceeds a preset threshold, it is determined that the paragraph is invalid content dominated by data, and the paragraph is removed. The third-level filtering strategy is a template matching strategy. Based on a pre-constructed document template library (including templates of common headers and footers, copyright declaration templates, table title templates, reference format templates, etc. in documents), a text similarity algorithm is used to calculate the similarity of a to-be-detected semantic paragraph and the text of the template library, and when the similarity exceeds a preset threshold, it is determined that the semantic paragraph is templated content. Such content is highly repetitive in different documents but has no actual originality, and is removed. After the three-level filtering, the remaining semantic paragraphs are valid semantic paragraphs for subsequent signature identification generation.

[0038] As shown in Figure 3 After obtaining the set of valid semantic paragraphs, each semantic paragraph in the set of semantic paragraphs can be subjected to segment division and mapping processing considering weights and positions, and a signature identification of each semantic paragraph is given, including: Each semantic paragraph in the set of semantic paragraphs is subjected to word segmentation processing to generate a word sequence of each semantic paragraph; Similarity calculation is performed on each word sequence, and adjacent words with a similarity reaching a predetermined threshold are combined to give a first semantic segment set; wherein, when performing similarity calculation, only adjacent words in the word sequence can be calculated; Each semantic paragraph in the set of semantic paragraphs is subjected to segment division based on a sliding window determination condition to give a second semantic segment set; The first semantic segment set and the second semantic segment set are combined and filtered to obtain a final semantic segment set; Based on a preset hash function family, each semantic segment in the final semantic segment set is subjected to identification and multiple fusion weight processing to give a signature identification of each semantic paragraph.

[0039] The word segmentation processing can use a lightweight model such as DistilBERT, similarity calculation is performed on each word sequence, adjacent words with a similarity reaching a predetermined threshold are combined to give a first semantic segment set, which can include: Each word in the word sequence is vectorized through a pre-trained language model to give a corresponding word vector; the pre-trained language model can be a FastText model; The word vectors of adjacent words in each word sequence are subjected to similarity calculation to obtain the similarity of the adjacent words; Adjacent words with a similarity exceeding a predetermined threshold are combined into a merged word; The merged word and the words that have not been merged are integrated as a semantic segment to give a first semantic segment set.

[0040] It can be understood that the language model such as FastText can learn the co-occurrence relationship between words through pre-training, and capture the context information of the words. For example, if the appearance frequency of "deep learning" and "model" is high, they are regarded as semantically related words, and the corresponding word vectors are adjusted to be closer during training, thereby improving the similarity between each other. Merge adjacent words with a similarity exceeding a predetermined threshold as a semantic segment, and all semantic segments and remaining words not merged form a first semantic segment set.

[0041] Specifically, the semantic paragraph is segmented, and a word sequence representation is generated as follows:

[0042] In the formula, W represents a word sequence, ω i (1≤i≤m) represents the i-th word in the word sequence.

[0043] For each word ω i , a corresponding word vector v is generated using the FastText pre-training model. Based on the word vectors of adjacent words, the similarity of adjacent words can be determined, which satisfies the following relationship:

[0044] In the formula, sim is the similarity, v i is the word vector of the i-th word, v i+1 is the word vector of the i+1-th word, v i and v i+1 represent the word vectors of adjacent words. For the determined similarity, when sim is greater than a predetermined threshold, adjacent words are merged into a semantic unit as a semantic segment, and all semantic segments and remaining words not merged form a first semantic segment set.

[0045] Based on the sliding window determination condition, each semantic paragraph in the semantic paragraph set is segmented to give a second semantic segment set, which can include: Determine the character sequence of each semantic paragraph; wherein the semantic paragraph itself is a character sequence composed of characters; Based on all character sequences, give the appearance probability of each character in each character sequence; Based on the characters possessed by each character sequence and in combination with the corresponding appearance probability, give the information entropy of each semantic paragraph; Based on the information entropy of each semantic paragraph, determine the corresponding sliding window size; Based on the corresponding sliding window size, segment each semantic paragraph to obtain a second semantic segment set.

[0046] In this example, the K-gram sliding window strategy is adopted to segment each semantic paragraph. The semantic paragraph is regarded as a character sequence T = [c1, c2,..., c n , where c n represents the nth character, and n is the total number of characters in the semantic paragraph. The size K of the sliding window is determined according to the information entropy of the semantic paragraph. The window step size is 1. By sliding the window, continuous character fragments, that is, K-gram fragments, are generated, specifically represented as {c i c i+1 ...c i+K+1 |1 ≤ i ≤ n - K + 1}. All the character fragments generated by the window sliding together constitute the second semantic fragment set of this semantic paragraph.

[0047] Among them, the size K of the sliding window is obtained through the information entropy, and the information entropy satisfies the following relationship:

[0048] In the formula, H is the information entropy of the semantic paragraph, c represents the characters in the semantic paragraph, chars represents the set of all characters in the semantic paragraph, and p(c) is the probability that the character c appears in the semantic paragraph.

[0049] Based on the information entropy, according to the preset K value selection strategy, the size K of the sliding window of the semantic paragraph is dynamically determined, specifically represented as:

[0050] In the formula, H ≥ 4 corresponds to high information entropy, indicating that the semantic paragraph has relatively complex content. When H < 3, it corresponds to low information entropy, indicating that the semantic paragraph has relatively simple content. By dynamically adjusting the size of the sliding window K, it can adapt to semantic paragraphs or text types of different complexities. For example, long terms in technical documents are retained in detail with a small sliding window, and short sentences in prose are resistant to noise with a large sliding window, realizing the optimization of the subsequent signature identification quality and improving the quality and efficiency of text compression.

[0051] The first semantic fragment set and the second semantic fragment set are merged to obtain a preliminary mixed set; the preliminary mixed set is filtered to obtain the final semantic fragment set. Specifically, filtering the preliminary mixed set can include complete deduplication, overlapping processing, and length filtering. Among them, complete deduplication is to remove exactly the same semantic fragments and only retain one such semantic fragment; overlapping processing is to retain both the semantic fragments with partial overlap in the second semantic fragment set and the first semantic fragment set, realizing the capture of local fragment details, so that the final semantic fragment set can retain the complete semantics; length filtering is to remove overly short semantic fragments (such as fragments with length < 2, such as remaining words without actual compression value like "in" and "zhong"), reducing invalid data and improving the efficiency of text information compression. For exampleFigure 4 As shown, the final semantic fragment set contains both K-gram fragments representing local features and semantic units with complete semantics, fully preserving the multidimensional information of the text to be compressed, and can also improve the accuracy of plagiarism detection in text plagiarism detection scenarios.

[0052] like Figure 5 As shown, based on a preset family of hash functions, the semantic segments of each semantic paragraph are identified and subjected to multiple fusion weight processing to provide a signature identifier for each semantic paragraph, which may include: The semantic segments of each semantic paragraph are weighted and fused using multiple weighting methods, and the fusion weight of each semantic segment is given. Each semantic fragment is concatenated with its location information to produce a location-enhanced semantic fragment; Based on each hash function with different seed values ​​in the preset hash function family, hash calculation is performed on all position-enhanced semantic fragments in each semantic segment, and the hash value of each position-enhanced semantic fragment corresponding to different hash functions is given; Based on the seed value of each hash function, the identifier of the text to be processed, and the fusion weight of each semantic segment, the adjustment value of different hash functions corresponding to the semantic enhancement segment at each position is given; The hash values ​​of the semantic augmentation fragments at each position in the semantic paragraph corresponding to different hash functions are fused with the corresponding adjustment values ​​to give the weighted hash values ​​of the semantic augmentation fragments at each position in the semantic paragraph corresponding to different hash functions. The minimum weighted hash value of each hash function is selected from all weighted hash values ​​of the semantic paragraph to provide a signature identifier for each semantic paragraph, specifically including: Select the minimum weighted hash value of each hash function from all weighted hash values ​​of the semantic paragraph to determine the signature component of each hash function in the semantic paragraph; Arrange all signature components in the semantic paragraph according to the order of the hash function to generate the signature identifier of the corresponding semantic paragraph.

[0053] In practical applications, signature generation converts valid semantic paragraphs into fixed-length digital signatures. Through multi-hash function calculations, it achieves high compression of text content while preserving text similarity features.

[0054] The semantic segments of each semantic paragraph are weighted using multiple methods such as TF-IDF weight, part-of-speech weight, and semantic weight. The weights obtained from each method are then fused together to determine the fused weight of each semantic segment.

[0055] Specifically, the TF-IDF weights are calculated for each semantic segment in the semantic segment set, satisfying the following relationship:

[0056] wherein, is the TF-IDF weight of the jth semantic segment, is the appearance frequency of the jth semantic segment s j , df(s j ) is the appearance frequency of the text containing the jth semantic segment , N is the total number of texts, which is the number of texts in the corpus, wherein the corpus can be pre-set according to actual scene requirements. The TF-IDF weight is based on the statistical information of the corpus, and is used to evaluate the importance of the semantic segment in the global context, is the inverse document frequency, which measures the scarcity of the semantic segment.

[0057] For the part-of-speech weight of each semantic segment in the semantic segment set, it can be determined based on a pre-set part-of-speech weight table. First, a dictionary or a part-of-speech tagging tool is used to label each semantic segment with a part of speech, such as using jieba, LTP or spaCy tagging tool, and then according to the labeled part of speech, the pre-set weight table is matched to assign a part-of-speech weight to each semantic segment. In this example, the part-of-speech weight of noun / professional term is 1.0, the part-of-speech weight of verb is 0.8, the part-of-speech weight of adjective is 0.6, and the part-of-speech weight of other words is 0.2.

[0058] The semantic weight refers to: if a semantic segment is a semantic unit (i.e., a semantic segment obtained by merging adjacent words in the semantic segment set), a fixed high weight such as 1.5 is assigned to it; or the TF-IDF weight, the part-of-speech weight of the semantic unit (semantic segment) is multiplied by an enhancement coefficient (such as 2) as the semantic weight. For non-semantic unit semantic segments, a fixed low weight is assigned to them; or the TF-IDF weight, the part-of-speech weight of the semantic segment is multiplied by a penalty coefficient (such as 0.5) as the semantic weight.

[0059] An attention mechanism is used to let the fusion model learn the importance of different weights, and the TF-IDF weight, the part-of-speech weight, and the semantic weight of each semantic segment are fused as the fusion weight. The fusion model is a Query-Key-Value (QKV) model, which calculates the attention score of each weight, then multiplies the attention score with the corresponding weight, and then sums up to complete the weight fusion to obtain the fusion weight of the semantic segment. The attention mechanism is specifically represented as:

[0060] In the formula, Q is a query, represents a weight importance, in this example, the mean of the TF-IDF weight, the part-of-speech weight, and the semantic weight after vectorization, K is a key, in this example, a splicing matrix of the TF-IDF weight, the part-of-speech weight, and the semantic weight, V is a value, in this example, a splicing matrix of the TF-IDF weight, the part-of-speech weight, and the semantic weight, and softmax is a normalization function.

[0061] The fusion coefficients of different weights can also be dynamically adjusted based on text attributes. For example, different fusion coefficients of the TF-IDF weight, the part-of-speech weight, and the semantic weight can be set according to text types such as academic texts, news texts, and novel texts, and different fusion coefficients of the TF-IDF weight, the part-of-speech weight, and the semantic weight can be set according to field attributes such as computer science, medicine, and finance. This makes the acquisition of the fusion weight more suitable for the characteristics of the text. For example, for academic texts, the coefficient of the semantic weight can be increased to focus on semantic fragments and scarce words, and for news texts, the coefficient of the TF-IDF weight can be increased to focus on event keywords and scarce words.

[0062] In this method, for the design of the hash function family, 128 independent MurmurHash3 hash functions are used to form the hash function family, and each hash function is generated using a different seed value. The seed value ranges from 1 to 2 32 -1, which can ensure the independence between hash functions. In the signature identification generation phase, for each semantic fragment s j ∈S in the semantic fragment set S corresponding to the semantic paragraph, the position encoding of the starting position p j of the semantic fragment is determined, that is, the character offset encoding within the semantic paragraph, and the position encoding is spliced into the semantic fragment to form a position-enhanced semantic fragment, which satisfies the following relationship:

[0063] In the formula, || represents splicing, encode() is an encoder, s j is a semantic fragment, p j is the starting position of the semantic fragment s j , and is a position-enhanced semantic fragment. The hash values of all position-enhanced semantic fragments under the 128 hash functions are calculated. The position-enhanced semantic fragment contains position information, which can enhance the readability and verifiability of the text compression result, facilitate auxiliary tracing, realize accurate positioning of repeated content in the text duplication detection scenario, avoid misjudgment of structural reorganization, and find local plagiarism.

[0064] According to the seed value of each hash function, the seed value of the random number generator is given, and a uniform random variable on [0, 1] is generated, combined with the fusion weight of each semantic segment, to give the adjustment value of the corresponding position-enhanced semantic segment. Specifically represented as:

[0065] In the formula, represents the adjustment value of the position-enhanced semantic segment , is the fusion weight of the semantic segment corresponding to the position-enhanced semantic segment, is a uniform random variable on [0, 1], generated by the seed value of the random number generator, and the seed value of the random number generator is obtained by the seed value of the hash function and the identification of the text to be processed. Specifically represented as:

[0066] In the formula, seed is the seed value of the random number generator, is the seed value of the i-th hash function, UUID is the identification of the text to be processed, which is the unique ID of the text to be processed, and M is a power of 2, which can be determined according to the actual situation.

[0067] The adjustment value of the position-enhanced semantic segment means that the core semantic segment weight is high, and its hash function value is more likely to become the smallest component of the signature identification; in the text duplication detection scene, it can improve the recognition of local plagiarism. Specifically, when copying the core semantic segment, the weighted hash value of its high-weight semantic segment will dominate the generation of the signature representation, and the Jaccard similarity calculation in the duplication detection is more accurate, which can avoid missing judgment.

[0068] Fuse the hash value of the position-enhanced semantic segment with the corresponding adjustment value to get the weighted hash value. Specifically represented as:

[0069] In the formula, is the weighted hash value of the position-enhanced semantic segment , is the hash value of the position-enhanced semantic segment , is the seed value of the i-th hash function, is the adjustment value of the position-enhanced semantic segment .

[0070] For each hash function h j , select the minimum value of all position-enhanced semantic segment weighted hash values under the hash function as the signature component, such as Figure 6The signature components are arranged in the order of the hash functions, and the signature identifier corresponding to the semantic paragraph is generated, which is specifically represented as:

[0071] M is the signature identifier of the semantic paragraph, is the minimum value of the weighted hash values of all position-enhanced semantic segments in the semantic segment set S under the i-th hash function, that is, the signature component. The 128 signature components are arranged in the order of the hash functions to form a 128-dimensional fixed-length signature identifier M = [m1, m2,..., m 128 ], wherein m i is the i-th signature component and m i = Whas{h i (s)|s∈S}, which significantly reduces the dimension of the representation when performing text compression and improves the compression efficiency.

[0072] As Figure 7 shown, after obtaining the signature identifier, the signature identifier can be indexed, and the signature identifier and the identification information of the semantic paragraph are combined to generate compressed text information for indexing the distributed database, including: The signature identifier is divided into multiple bands based on a preset division strategy; The corresponding byte sequence of each band is calculated, and the bucket key corresponding to each band is given; The signature identifier, the corresponding text to be processed of the signature identifier, and the identification information of the semantic paragraph corresponding to the signature identifier are stored in the index distributed database using the bucket key as the index.

[0073] In actual application scenarios, the LSH (Local Sensitivity Hashing) bucketing strategy is used to divide the bands. The LSH bucketing strategy classifies and stores the MinHash signature, which can quickly aggregate similar signature identifiers. At the same time, a distributed storage architecture is adopted to meet the efficient storage and query requirements of large-scale signature data. Specifically, the following steps are included: The generated 128-dimensional MinHash signature (i.e., the signature identifier of the semantic paragraph) M = [m1, m2,..., m 128 ] is taken as the processing object, wherein the signature component m j is divided into multiple bands using the LSH bucketing strategy. The 128-dimensional MinHash signature is evenly divided into multiple bands band according to the continuous dimensions, and each band contains multiple signature components. For example, the 128-dimensional MinHash signature is evenly divided into 16 bands band according to the continuous dimensions, and each band contains 8 signature components, i.e., the band index band t The corresponding signature component interval is [m 8(t-1)+1 , m 8(t-1)+2 ,..., m8t ], band t is the t-th band, band t = {m k | k = 8(t - 1) + 1, 8(t - 1) + 2,..., 8t}, t is the band index, 1≤t≤16, m k is the k-th component in the MinHash signature, 8 is the number of signature components contained in each band, and for each band band t , its contained 8 signature components are extracted to form a sub-vector, and the MD5 hash algorithm is used to operate the sub-vector to generate a bucket key, and the sub-vector [m 8(t-1)+1 , m 8(t-1)+2 ,..., m 8t ] is converted into a byte sequence, each signature component is encoded as a 4-byte unsigned integer, and a 128-bit hash value is obtained as the bucket key BK t corresponding to the band, that is, BK t == MD5((bytes(m k )) = MD5(bytes(m 8(t-1)+1 ) || bytes(m 8(t-1)+2 ) ||... || bytes(m 8t )), where bytes(m k ) represents the operation of converting the k-th signature component into a byte sequence, || represents the splicing operation of the byte sequence, and BK t is the bucket key corresponding to the t-th band.

[0074] After determining the bucket key corresponding to each wave band, a collaborative architecture of a real-time cache database and a full-amount storage database is used to store the bucket key, the signature identifier, the signature identifier corresponding to the to-be-processed text, and the identification information of the signature identifier corresponding to the semantic paragraph. The real-time cache database stores the mapping relationship between the bucket key and the MinHash signature, that is, after the bucket key is determined, the corresponding MinHash signature can be read from the real-time cache database. The full-amount storage database stores the bucket key, the MinHash signature, and the mapping relationship therebetween, and also stores full-amount data such as the to-be-processed text corresponding to the signature identifier and the identification information of the semantic paragraph corresponding to the signature identifier. In an actual application scenario, a distributed database architecture in which Redis (Remote Dictionary Server) and HBase (Hadoop Database) are collaborative can be used to store the MinHash signature and the original text and paragraph identification information corresponding thereto in a database indexed by the bucket key, so as to realize efficient data management of the cache and the full-amount storage. Redis, as a cache database, is used to store the mapping relationship between the bucket key and the MinHash signature, each bucket key is taken as a key, and the corresponding MinHash signature M is taken as a value, so as to construct a key-value pair storage. Since Redis is based on memory storage, it can provide a millisecond-level query response speed and can meet the rapid candidate recall requirement in a high-frequency query scenario. HBase, as a distributed column storage database, is used to store full-amount data to ensure long-term reliability and scalability of the data. The table structure is designed as follows: the row key is set as the bucket key to ensure centralized storage of data with the same bucket key; the column family is divided into three categories, in which the sig column family is used to store the MinHash signature M (stored in the form of binary or string), the doc column family is used to store the identification information of the text (including a text unique identifier, a text name, an upload time, a file format, and the like), and the para column family is used to store the semantic paragraph identification information (including the starting position and the ending position of the semantic paragraph in the text, a paragraph serial number, and the like). This column family type storage structure supports flexible field extension and facilitates fast query of the complete document and paragraph information corresponding to the signature according to the bucket key. In the data writing process, the bucket key, the MinHash signature, and the identification information are written into the Redis cache and the HBase full-amount database synchronously to ensure consistency of the cache and the full-amount data. When the cache data in Redis exceeds the preset validity period or storage capacity, the cache is evicted through the LRU (Least Recently Used) strategy. The evicted data can be read from HBase again to ensure the availability of the data.

[0075] As shown in Figure 8 After generating the compressed text information for indexing the distributed database, the present application can further include: The generated compressed text information is used to process the to-be-duplicated text (for example, the to-be-duplicated text is sequentially preprocessed, multi-level semantic paragraph division, segment division, mapping processing, and indexing processing), and the signature identification and indexing processing result of the to-be-duplicated text are given. Based on the indexing processing result of the to-be-duplicated text, the corresponding signature identification is queried from the index distributed database. Based on the signature identification of the to-be-duplicated text and the signature identification of the index distributed database, the text duplication result is given.

[0076] In the process of text information duplication checking, the inputted text to be checked (which can be a single text or a paragraph) is preprocessed, multi-level semantic paragraph division, segment division and mapping processing, and indexing processing are repeatedly performed, specifically, the text to be checked is format-converted and noise-removed to obtain a standardized pure text; semantic segmentation and invalid paragraph filtering are performed to obtain effective semantic paragraphs of the text to be checked; K-gram processing, hash function calculation and MinHash signature generation are performed on each effective paragraph to obtain a MinHash signature set of the text to be checked; and each MinHash signature is divided into multiple wave segments according to the LSH bucketing strategy to generate a corresponding bucket key set. To improve the candidate recall efficiency, the BloomFilter is used to pre-filter the bucket key set of the text to be checked, specifically, for each bucket key of the text to be checked, the BloomFilter is first queried, if the BloomFilter determines that the bucket key does not exist, the subsequent query of the bucket key is directly skipped; if it is determined to exist, the MinHash signature corresponding to the bucket key is queried from the distributed database. In the process of query, Redis cache is first queried, if the MinHash signature corresponding to the bucket key exists in Redis, the MinHash signature is directly read; if Redis is not hit (i.e. there is no bucket key data in the cache), the HBase full database is queried to read the MinHash signature corresponding to the bucket key; then all the MinHash signatures queried from Redis and HBase are de-duplicated, i.e. the repeated signatures are removed to avoid subsequent repeated calculation, and finally a candidate signature set of the text to be checked is obtained. The Jaccard similarity is calculated between the query MinHash signature of the text to be checked and the candidate MinHash signature queried from the index distributed database, i.e. Jaccard(A,B)=|A∩B| / |A∪B|, wherein A is a semantic segment set corresponding to the query MinHash signature, B is a semantic segment set corresponding to the candidate MinHash signature, and the similarity value is obtained by calculating the size ratio of the intersection and union of the two semantic segment sets. When the Jaccard similarity of the query MinHash signature of the text to be checked and the candidate MinHash signature is greater than or equal to a first threshold value, the position information of the repeated semantic paragraph is recorded, including the UUID of the text, the starting position of the semantic paragraph, the repeated length, etc.; when the Jaccard similarity is less than the first threshold value and greater than or equal to a second threshold value, it is determined as a suspected duplicate text, and a review mark is performed, which needs to be manually reviewed to realize secondary review to determine whether it is a duplicate text; when the Jaccard similarity is less than the second threshold value, it is determined as a non-duplicate text. The first threshold value is in the range of 0.8-0.9, preferably 0.85, and the second threshold value is in the range of 0.5-0.7, preferably 0.6.After the judgment based on the first threshold value and the second threshold value, the proportion of the repeated text and the suspected repeated text in the to-be-checked repeated text can be calculated respectively, and the repeated text report containing the similarity value, the repeated position and the proportion information can be generated by sorting from high to low according to the proportion, so as to facilitate subsequent rapid positioning of the repeated content.

[0077] After the data storage and the text duplication checking are implemented, the duplication checking result can be written into a transaction database, an analysis database and a search engine index synchronously, multi-dimensional query and analysis are supported, meanwhile, the whole process data is recorded through a log collection link, the performance indicators such as the false positive rate and the false negative rate are monitored in real time, when the performance indicators deviate from the predetermined range, the duplication checking result is manually audited, the misjudgment cases are collected, the seed value of the MinHash hash function and the band number of the LSH bucket are adjusted based on the misjudgment cases, that is, when the false positive rate > 5% or the false negative rate > 5%, an alarm is triggered to start an offline training process, the seed value of the MinHash hash function and the LSH bucket parameter are optimized and updated, the band number is increased from 16 to 20, so as to reduce the number of candidate signatures in the bucket, or the band number is reduced from 16 to 12, so as to expand the recall range of the candidate signatures, and the updated parameter is deployed to the online system after testing.

[0078] As shown in Figure 9 The present application also provides a text information compression device for indexing a distributed database, which adopts the above-mentioned text information compression method, and the device comprises: a preprocessing module for obtaining to-be-processed text and performing preprocessing to give pure text in a predetermined format; a paragraph division module for performing multi-level semantic paragraph division on the pure text in the predetermined format to give a corresponding semantic paragraph set; an identification generation module for performing fragment division and mapping processing on each semantic paragraph in the semantic paragraph set considering weight and position to give a signature identification of each semantic paragraph; an indexing and storage module for performing indexing processing on the signature identification, and combining the signature identification and the identification information of the semantic paragraph to generate compressed text information for indexing the distributed database.

[0079] Further, the preprocessing module is used for: obtaining multi-source to-be-processed text; performing unified coding on the multi-source to-be-processed text to generate initial text; performing cleaning and denoising processing on the initial text to obtain pure text in a predetermined format.

[0080] Further, the paragraph division module is used for: punctuate and line space the plain text in the predetermined format, determine the position and type of all punctuation in the plain text, and the position of line space formed by two or more consecutive line spaces; determine the forced segmentation point based on the position and type of all punctuation in the plain text, and the position of line space formed by two or more consecutive line spaces; cut the plain text in the predetermined format once according to the forced segmentation point, to obtain a plurality of basic text segments; process and analyze the basic text segments through a sliding window, to give a cosine similarity matrix; determine the cosine similarity change rate of a plurality of consecutive sliding windows based on the cosine similarity matrix, and give a semantic break point in combination with a preset amplitude threshold; cut the plurality of basic text segments twice based on the semantic break point, to give an initial semantic paragraph set; filter the initial semantic paragraph set based on a preset filtering strategy, to give a final semantic paragraph set.

[0081] Further, the identification generation module is configured to: perform word segmentation processing on each semantic paragraph in the semantic paragraph set, to generate a word sequence of each semantic paragraph; perform similarity calculation on each word sequence, and combine adjacent words with a similarity reaching a predetermined threshold, to give a first semantic segment set; perform segment division on each semantic paragraph in the semantic paragraph set based on a sliding window determination condition, to give a second semantic segment set; mix and filter the first semantic segment set and the second semantic segment set, to obtain a final semantic segment set; perform identification and multiple fusion weight processing on each semantic segment in the final semantic segment set based on a preset hash function family, to give a signature identification of each semantic paragraph.

[0082] Further, the identification generation module is configured to: vectorize each word sequence through a pre-trained language model, to give a word vector of each word in the word sequence; perform similarity calculation on the word vectors of adjacent words in each word sequence, to obtain the similarity of adjacent words; combine adjacent words with a similarity exceeding a predetermined threshold into a merged word; integrate the merged word and the words that are not merged as a semantic segment, to obtain a first semantic segment set.

[0083] Further, the identification generation module is configured to: determine a character sequence of each semantic paragraph; Based on all character sequences, the occurrence probability of each character in each character sequence is given respectively; Based on the characters possessed by each character sequence and the corresponding occurrence probability, the information entropy of each semantic paragraph is given; Based on the information entropy of each semantic paragraph, the corresponding sliding window size is determined; Based on the corresponding sliding window size, the semantic paragraph is segmented to obtain a second semantic segment set.

[0084] Further, the identification generation module is further used to: The semantic segment of each semantic paragraph is calculated and fused by a plurality of weight calculation methods respectively to give the fusion weight of each semantic segment; Each semantic segment and its position information are spliced to give a position-enhanced semantic segment; Based on each hash function in the preset hash function family using different seed values, all position-enhanced semantic segments in each semantic paragraph are calculated by hash to give the hash value of each position-enhanced semantic segment corresponding to different hash functions; Based on the seed value of each hash function, the identification of the text to be processed and the fusion weight of each semantic segment, the adjustment value of each position-enhanced semantic segment corresponding to different hash functions is given; The hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions is fused with the corresponding adjustment value to give the weighted hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions. From all the weighted hash values of the semantic paragraph, the minimum weighted hash value of each hash function is selected to give the signature identification of each semantic paragraph.

[0085] Further, the index and storage module is used to: Based on the preset division strategy, the signature identification is divided into a plurality of wave bands; The corresponding byte sequence of each wave band is calculated respectively to give the bucket key corresponding to each wave band; The bucket key is used as an index to store the signature identification, the identification information of the text to be processed corresponding to the signature identification and the semantic paragraph corresponding to the signature identification to the index distributed database.

[0086] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the foregoing description without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be interpreted as including all such variations and modifications as fall within the spirit and scope of the application. It is further intended that the disclosure of all such modifications and variations be included within the scope of the application, the terms used herein being defined solely for purposes of the description being applied thereto unless otherwise indicated.

Claims

1. A method for compressing text information for indexing a distributed database, characterized in that, include: Get the text to be processed and preprocess it to give plain text in a predefined format; Perform multi-level semantic paragraph division on plain text with a predefined format and provide the corresponding set of semantic paragraphs; For each semantic segment in the semantic segment set, perform segmentation and mapping processing that takes into account weight and position, and give a signature identifier for each semantic segment; The signature identifier is indexed and combined with the identifier information of the semantic paragraph to generate compressed text information for indexing the distributed database.

2. The text information compression method as described in claim 1, characterized in that, Obtain the text to be processed and preprocess it to provide plain text in a predefined format, including: Retrieve text from multiple sources; The texts from multiple sources to be processed are uniformly encoded to generate the initial text; The initial text is cleaned and denoised to obtain plain text in a predetermined format.

3. The text information compression method as described in claim 1, characterized in that, Perform multi-level semantic paragraph division on plain text with a predefined format, and provide the corresponding set of semantic paragraphs, including: Punctuation and blank lines are identified in plain text with a predefined format. The position and type of all punctuation marks in the plain text are determined, as well as the position of blank lines formed by two or more consecutive line breaks. The forced breakpoint is determined based on the position and type of all punctuation marks in the plain text, as well as the position of the blank line formed by two or more consecutive newline characters. Based on the forced segmentation points, the plain text in the predefined format is cut into multiple basic text segments. The basic text fragments are processed and analyzed using a sliding window, and a cosine similarity matrix is ​​given. Based on the cosine similarity matrix, the cosine similarity change rate of multiple consecutive sliding windows is determined, and semantic breakpoints are given in combination with a preset amplitude threshold. Based on semantic breakpoints, multiple basic text segments are subjected to secondary text segmentation to give an initial set of semantic paragraphs; The initial set of semantic paragraphs is filtered based on a preset filtering strategy to give the final set of semantic paragraphs.

4. The text information compression method according to any one of claims 1-3, characterized in that, For each semantic segment in the semantic segment set, perform segmentation and mapping processing taking into account weight and position, and give a signature identifier for each semantic segment, including: Each semantic segment in the semantic segment set is segmented into words to generate a word sequence for each semantic segment; Calculate the similarity of each word sequence, merge adjacent words whose similarity reaches a predetermined threshold, and give the first set of semantic segments; Based on the sliding window determination conditions, each semantic segment in the semantic segment set is divided into segments to give a second set of semantic segments; The first and second semantic fragment sets are merged and filtered to obtain the final semantic fragment set; Based on a pre-defined family of hash functions, each semantic segment in the final semantic segment set is identified and subjected to multiple fusion weighting processes to provide a signature identifier for each semantic segment.

5. The text information compression method as described in claim 4, characterized in that, For each word sequence, similarity is calculated, and adjacent words with similarity reaching a predetermined threshold are merged to give the first semantic fragment set, including: Each word sequence is vectorized using a pre-trained language model, resulting in a word vector for each word in the sequence. The similarity between adjacent words is calculated by performing word vector similarity calculation on the word vectors of adjacent words in each word sequence. Combine adjacent words with a similarity exceeding a predetermined threshold into a single merged word; The merged words and the unmerged words are treated as semantic segments and then integrated to obtain the first set of semantic segments.

6. The text information compression method as described in claim 5, characterized in that, Based on the sliding window determination conditions, each semantic segment in the semantic segment set is divided into segments, resulting in a second set of semantic segments, including: Determine the character sequence for each semantic segment; Based on all character sequences, give the probability of each character appearing in each character sequence; Based on the characters in each character sequence and their corresponding occurrence probabilities, the information entropy of each semantic segment is given; The size of the corresponding sliding window is determined based on the information entropy of each semantic segment. Each semantic segment is divided into segments based on the corresponding sliding window size to obtain a second set of semantic segments.

7. The text information compression method as described in claim 4, characterized in that, Based on a pre-defined family of hash functions, the semantic segments of each semantic paragraph are identified and subjected to multiple fusion weighting processes to provide a signature identifier for each semantic paragraph, including: The semantic segments of each semantic paragraph are weighted and fused using multiple weighting methods, and the fusion weight of each semantic segment is given. Each semantic fragment is concatenated with its location information to produce a location-enhanced semantic fragment; Based on each hash function with different seed values ​​in the preset hash function family, hash calculation is performed on all position-enhanced semantic fragments in each semantic segment, and the hash value of each position-enhanced semantic fragment corresponding to different hash functions is given; Based on the seed value of each hash function, the identifier of the text to be processed, and the fusion weight of each semantic segment, the adjustment value of different hash functions corresponding to the semantic enhancement segment at each position is given; The hash values ​​of the semantic augmentation fragments at each position in the semantic paragraph corresponding to different hash functions are fused with the corresponding adjustment values ​​to give the weighted hash values ​​of the semantic augmentation fragments at each position in the semantic paragraph corresponding to different hash functions. Select the minimum weighted hash value of each hash function from all weighted hash values ​​of the semantic paragraph to give the signature identifier of each semantic paragraph.

8. The text information compression method as described in claim 7, characterized in that, The signature identifier is indexed, and combined with the signature identifier and the identifier information of semantic paragraphs, compressed text information for indexing the distributed database is generated, including: Based on a preset partitioning strategy, the signature identifier is divided into multiple bands; Calculate the corresponding byte sequence for each band and give the bucket key for each band; Using bucket keys as indexes, the signature identifier, the text to be processed corresponding to the signature identifier, and the identifier information of the semantic paragraph corresponding to the signature identifier are stored in the indexed distributed database.

9. The text information compression method as described in claim 7, characterized in that, Also includes: The text to be checked for plagiarism is preprocessed, divided into multi-level semantic paragraphs, fragments, mapped, and indexed in sequence, and the signature identifier and indexing result of the text to be checked for plagiarism are given. Based on the indexing results of the text to be deduplicated, the corresponding signature identifier is queried from the indexed distributed database; Based on the signature identifier of the text to be checked and the signature identifier of the indexed distributed database, the text plagiarism detection results are given.

10. A text information compression device for indexing a distributed database, characterized in that, The apparatus employing the text information compression method as described in any one of claims 1 to 9 includes: The preprocessing module is used to acquire the text to be processed and preprocess it to give plain text in a predetermined format; The paragraph segmentation module is used to perform multi-level semantic paragraph segmentation on plain text with a predefined format and provide the corresponding set of semantic paragraphs; The identifier generation module is used to perform segmentation and mapping processing on each semantic segment in the semantic segment set, taking into account weight and position, and to give a signature identifier for each semantic segment. The indexing and storage module is used to index the signature identifier and, in combination with the signature identifier and the identifier information of the semantic paragraph, generate compressed text information for indexing the distributed database.

Images