LLM2Vec algorithm-based text duplicate checking method and system
The LLM2Vec algorithm converts text into feature vectors and uses cosine similarity, combined with bidirectional language model and dropout mask, solves the problem of insufficient semantic understanding in the existing technology, improves the accuracy of text mutation checking and the understanding of complex semantics ability.
Patent Information
- Application Number
- CN202410954853.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-01
AI Technical Summary
The existing text mutation checking method lacks semantic understanding of text content, resulting in low accuracy of mutation checking. In particular, algorithms based on string matching are easily affected by text rewriting and synonym replacement, while TF-IDF-based methods are difficult to capture the deep layer. Semantic information.
Using the text mutation check method based on the LLM2Vec algorithm, the text is converted into an eigenvector and the cosine similarity method is used to calculate the similarity between the eigenvectors. Combining the bidirectional language model and dropout mask to generate high-quality text representations, capturing the text rich semantic information and distinguish nuances of different texts.
It improves the accuracy of text plagiarism checking, can better distinguish similar and dissimilar texts, enhances the ability to understand complex language structures and semantics, and is suitable for tasks such as text clustering and similarity search.
Smart Images

Figure CN118885602A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text duplicate checking, and in particular, to a text duplicate checking method and system based on an LLM2Vec algorithm. Background Art
[0002] In the field of information processing, text usually refers to a data sequence composed of characters. These characters can be letters, numbers, punctuation marks and other symbols. They are combined into sentences and paragraphs according to certain grammatical and semantic rules, and finally form readable articles, reports, books, etc. Text data is one of the main forms of human communication and information recording.
[0003] Text duplication checking can identify plagiarism and ensure that the author's intellectual property is protected, while ensuring the integrity of academic research and publishing. In the field of content creation, such as news and blog writing, duplication checking can ensure the uniqueness and novelty of the content, enhance the brand's reputation and readers' trust. In educational institutions, teachers use duplication checking tools to evaluate students' homework and papers to ensure that the work submitted by students is completed independently, thereby fairly evaluating students' learning outcomes.
[0004] Existing text duplication checking methods mainly include algorithms based on string matching, algorithms based on natural language processing, etc. String matching algorithms, such as the longest common subsequence (LCS) algorithm, identify plagiarism by directly comparing the similarity of text strings. Natural language processing-based algorithms, such as the TF-IDF (term frequency-inverse document frequency) method, evaluate similarity by analyzing the word frequency distribution in the text.
[0005] The defects of the existing technology mainly lie in the insufficient semantic understanding of the text content, resulting in low accuracy of duplicate detection. For example, the algorithm based on string matching is easily affected by means such as text rewriting and synonym replacement, while the TF-IDF-based method is difficult to capture the deep semantic information of the text.
[0006] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention
[0007] In order to overcome the above problems, the present invention aims to propose a text duplication checking method and system based on the LLM2Vec algorithm, with the aim of solving the problem that the defects of the prior art are mainly insufficient semantic understanding of the text content, resulting in low accuracy of duplication checking.
[0008] To this end, the specific technical solution adopted by the present invention is as follows:
[0009] According to one aspect of the present invention, a text duplication checking method based on the LLM2Vec algorithm is provided. The text duplication checking method and system based on the LLM2Vec algorithm include the following steps:
[0010] S1. Use LLM2Vec algorithm to convert text into feature vector;
[0011] S2, using the cosine similarity method to calculate the cosine similarity between each feature vector;
[0012] S3. Preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and determine whether the articles are similar.
[0013] Optionally, converting text into feature vectors using the LLM2Vec algorithm includes the following steps:
[0014] S11, by replacing the causal attention mask with an all-one matrix and converting the decoder language-only model to a bidirectional model;
[0015] S12, training the bidirectional model and optimizing the weight of the bidirectional model;
[0016] S13, forward propagating the same input sequence twice through the optimized bidirectional model through the dropout mask, obtaining two different representations of the same sequence, and maximizing the similarity of the two representations to obtain the feature representation;
[0017] S14. Input the text into a text encoder and convert it into a feature vector using the text encoder.
[0018] Optionally, by replacing the causal attention mask with an all-one matrix, converting the decoder to a bidirectional model includes the following steps:
[0019] S111, create a matrix with the same size as the attention matrix, and set all elements in the attention matrix to 1;
[0020] S112. Based on the set attention matrix, in the attention layer of the decoder language model only, the causal attention mask is replaced with an all-1 matrix.
[0021] Optionally, forward propagating the same input sequence twice through the optimized bidirectional model through a dropout mask to obtain two different representations of the same sequence and maximizing the similarity of the two representations to obtain a feature representation includes the following steps:
[0022] S131, generating a dropout mask for the input sequence;
[0023] S132, using the dropout mask to perform two independent forward propagations on the input sequence, and in each propagation, discarding or retaining the corresponding neuron output according to the dropout mask;
[0024] S133, two forward propagations obtain two different representations of the same input sequence;
[0025] S134. Maximize the similarity between the two representations to obtain feature representation.
[0026] Optionally, calculating the cosine similarity between the feature vectors using the cosine similarity method includes the following steps:
[0027] S21, constructing a vector space model, and inputting two feature vectors into the vector space model;
[0028] S22. In the vector space model, the cosine value of the angle between two eigenvectors is calculated using the cosine similarity method;
[0029] S23. Determine the similarity between the two feature vectors based on the cosine value of the angle between the two feature vectors.
[0030] Optionally, the formula for calculating the cosine value of the angle between two feature vectors using the cosine similarity method is:
[0031]
[0032] sim(v1,v2) represents the similarity between feature vectors V1 and V2;
[0033] V1 and V2 represent the feature vectors respectively;
[0034] || || represents the length of the feature vector;
[0035] · represents the dot product of eigenvectors.
[0036] Optionally, the preset similarity threshold is 0.8.
[0037] Optionally, a similarity threshold is preset, and the calculated cosine similarity is compared with the similarity threshold, and determining whether the articles are similar includes the following steps:
[0038] If the calculated cosine similarity exceeds 0.8, the text is judged to be similar text;
[0039] If the calculated cosine similarity is lower than 0.8, the texts are judged to be dissimilar.
[0040] According to another aspect of the present invention, there is also provided a text duplicate checking system based on the LLM2Vec algorithm, the text duplicate checking system comprising: an LLM2Vec feature extraction module, a cosine similarity calculation module and a similarity judgment module;
[0041] LLM2Vec feature extraction module, used to convert text into feature vectors using the LLM2Vec algorithm;
[0042] A cosine similarity calculation module is used to calculate the cosine similarity between each feature vector using a cosine similarity method;
[0043] The similarity judgment module is used to preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and judge whether the articles are similar.
[0044] Optionally, the LLM2Vec feature extraction module includes: a bidirectional model building module, a bidirectional model training module, a feature representation learning module and a text encoder module;
[0045] A bidirectional model building module for converting a decoder-only language model to a bidirectional model by replacing the causal attention mask with an all-ones matrix;
[0046] The bidirectional model training module is used to train the bidirectional model and optimize the weight of the bidirectional model;
[0047] The feature representation learning module is used to forward propagate the same input sequence twice through the optimized bidirectional model through the dropout mask to obtain two different representations of the same sequence and maximize the similarity of the two representations to obtain the feature representation;
[0048] The text encoder module is used to input the text into the text encoder and convert it into a feature vector using the text encoder.
[0049] Compared with the prior art, this application has the following beneficial effects:
[0050] 1. The present invention modifies the attention mechanism of the language model from unidirectional (considering only the previous token) to bidirectional (considering both the previous and next tokens). This enables the representation of each token to contain information about the entire sequence, thereby generating a richer and more comprehensive text representation. The bidirectional attention mechanism helps capture long-distance dependencies in the text, which is critical for understanding complex language structures and semantics. In unsupervised contrastive learning, the model generates two different representations of the same input sequence through a dropout mask, and then trains the model through a contrastive learning method so that the two representations are as similar as possible and as different as possible from the representations of other sequences. This method helps the model learn the ability to distinguish between different texts, thereby generating high-quality text representations that can capture subtle differences between texts. In this way, the model is able to better distinguish between similar and dissimilar texts, which is very useful for tasks such as text clustering and similarity search.
[0051] 2. The text encoder of the present invention can not only capture the rich semantic information of the text but also distinguish the subtle differences between different texts by generating high-quality text vector representations, which is very valuable for various natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The above characteristics, features and advantages of the present invention and their implementation methods and methods will become more clearly understood in conjunction with the following description of the embodiments, which are described in detail in conjunction with the accompanying drawings. Herein, a schematic diagram is shown:
[0053] Figure 1 is a flowchart of a text duplication checking method based on the LLM2Vec algorithm according to an embodiment of the present invention;
[0054] Figure 2 It is a principle block diagram of a text duplication checking system based on the LLM2Vec algorithm according to an embodiment of the present invention.
[0055] In the figure:
[0056] 1. LLM2Vec feature extraction module; 2. Cosine similarity calculation module; 3. Similarity judgment module. DETAILED DESCRIPTION
[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0058] According to an embodiment of the present invention, a text duplication checking method and system based on the LLM2Vec algorithm are provided.
[0059] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to one embodiment of the present invention, a text duplicate checking method based on the LLM2Vec algorithm is provided, and the text duplicate checking method and system based on the LLM2Vec algorithm include the following steps:
[0060] S1. Use the LLM2Vec algorithm to convert text into feature vectors.
[0061] Preferably, converting text into a feature vector using the LLM2Vec algorithm comprises the following steps:
[0062] S11, by replacing the causal attention mask with an all-one matrix and converting the decoder language-only model to a bidirectional model;
[0063] S12, training the bidirectional model and optimizing the weight of the bidirectional model;
[0064] It should be noted that training the bidirectional model and optimizing the weight of the bidirectional model include the following steps:
[0065] 1) Define the loss function: First, you need to define a loss function to measure the difference between the output predicted by the model and the true label. For different tasks, the loss function may be different. For example, the classification task may use the cross entropy loss, and the regression task may use the mean square error loss.
[0066] 2) Forward propagation: During the training process, the input data is forward propagated through the model to obtain the predicted output.
[0067] 3) Calculate gradients: Use the loss function to calculate the error between the predicted output and the true label, and use the back-propagation algorithm to calculate the gradient of each weight. The gradient indicates how to adjust the weight to reduce the loss.
[0068] 4) Apply an optimization algorithm: Use gradient descent to update the weights to minimize the loss function.
[0069] 5) Adjust hyperparameters: During the training process, you may need to adjust hyperparameters such as learning rate, batch size, regularization parameters, etc. to optimize model performance.
[0070] 6) Regularization: To prevent overfitting, L2 regularization and Dropout techniques are used to limit the model complexity.
[0071] S13, through the dropout mask, the same input sequence, where the input sequence is the digital expression of the text after embedding, is forward propagated twice through the optimized bidirectional model to obtain two different representations of the same sequence, and the similarity of the two representations is maximized to obtain the feature representation, and the feature representation is the text feature representation obtained by the optimized model. The maximization of the two representations above is the optimization of the model;
[0072] S14. Input the text into a text encoder and convert it into a feature vector using the text encoder.
[0073] In addition, it needs to be explained that the bidirectional model refers to the deep learning model of the present application; the bidirectional attention mechanism is a mechanism used in natural language processing (NLP) to enhance the model's ability to understand text sequences, which can be understood as a specific step in the model to process data.
[0074] Preferably, by replacing the causal attention mask with an all-one matrix, converting the decoder to a bidirectional model comprises the following steps:
[0075] S111, create a matrix with the same size as the attention matrix, and set all elements in the attention matrix to 1;
[0076] S112. Based on the set attention matrix, in the attention layer of the decoder-only language model, the causal attention mask is replaced with an all-1 matrix, where the attention layer refers to a part of the structure in the model. The attention matrix is the actual result calculated by this part of the structure.
[0077] Preferably, forward propagating the same input sequence twice through the optimized bidirectional model through the dropout mask to obtain two different representations of the same sequence and maximizing the similarity of the two representations to obtain the feature representation includes the following steps:
[0078] S131, generate a dropout mask for the input sequence, the elements in the mask are random 0 and 1, 0 means that the neuron output at this position will be discarded in the forward propagation, and 1 means it will be retained;
[0079] S132. Perform two independent forward propagations on the input sequence using the dropout mask. During each propagation, the corresponding neuron output is discarded or retained according to the dropout mask. In addition, a neuron is a set of mathematical formulas in a deep learning model; for example, softmax or other mathematical formulas; the input sequence is used as an input parameter to enter the neuron and the mathematical formula for calculation.
[0080] S133, two forward propagations obtain two different representations of the same input sequence;
[0081] S134. Maximize the similarity between the two representations to obtain feature representation.
[0082] As a note, traditionally, decoder-only language models (like GPT) use a causal attention mechanism, meaning that the representation of each token is only influenced by the previous tokens.
[0083] LLM2Vec first converts the decoder into a bidirectional model by replacing the causal attention mask with an all-ones matrix, allowing each token in the model to access all other tokens in the sequence.
[0084] A bidirectional model is a model that can consider the context of the input sequence at the same time in natural language processing (NLP). In traditional language models, such as GPT (Generative Pre-trained Transformer), a causal attention mechanism is used, which means that when generating the representation of each token, only all the tokens before it can be considered, and the tokens after it cannot be accessed. This mechanism is often used in text generation tasks because text generation needs to be built step by step from front to back. The bidirectional model allows the model to consider the contextual information on the left and right sides of each token at the same time when processing it, thereby improving the model's ability to understand the entire sequence.
[0085] The all-ones matrix here refers to a matrix whose all elements are set to 1. In the LLM2Vec algorithm, the all-ones matrix is used to replace the traditional causal attention mask. In natural language processing, the attention mask is used to control the visibility in the attention mechanism, that is, which tokens can affect which tokens.
[0086] In the LLM2Vec algorithm, by replacing the causal attention mask with an all-1 matrix, each token in the model is able to access all other tokens in the sequence. This means that the model is no longer limited to only seeing the previous token, but can see both the previous and the next tokens at the same time, realizing a bidirectional attention mechanism. Specifically, this can be achieved by following the steps below:
[0087] 1) Initialize the all-1 matrix: Create a matrix of the same size as the attention matrix and set all elements to 1.
[0088] 2) Replace the attention mask: In the attention layer of the model, the original causal attention mask is replaced with an all-1 matrix.
[0089] 3) Adjust model weights: Since the model now has access to bidirectional information, the model weights may need to be adjusted to accommodate the new bidirectional attention mechanism.
[0090] 4) Training or fine-tuning the model: Models using all-1 matrices may require further training or fine-tuning to optimize their performance on specific tasks.
[0091] In this way, the LLM2Vec algorithm converts the original decoder-only language model into a model that can process bidirectional information, thereby improving the model's ability to understand text and the performance of the task.
[0092] Dropout mask is part of the dropout technique in machine learning. Dropout is a regularization technique used to prevent neural networks from overfitting. During training, dropout randomly sets some neurons in the network (or the output of neurons) to zero, so that the network is slightly different each time it is trained, enhancing the generalization ability of the model. The dropout mask is a random mask used to determine which neurons' outputs will be set to zero.
[0093] The same input sequence is forward-propagated through the model twice with independently sampled dropout masks to obtain two different representations of the same sequence, and then the model is trained to maximize the similarity between these two representations while reducing the similarity with other sequence representations.
[0094] By optimizing the model parameters, the similarity between the two representations of the same input sequence is maximized. At the same time, the model also needs to learn to reduce the similarity with the representations of other different input sequences, which is usually achieved through a contrast loss function. After training, the optimized model parameters and learned feature representations are obtained. These feature representations can be used as text encodings for various downstream tasks, such as text similarity measurement, clustering, retrieval, etc. This is the so-called "text encoder".
[0095] S2. Calculate the cosine similarity between each feature vector using the cosine similarity method.
[0096] Preferably, calculating the cosine similarity between the feature vectors using the cosine similarity method comprises the following steps:
[0097] S21, constructing a vector space model, and inputting two feature vectors into the vector space model;
[0098] S22. In the vector space model, the cosine value of the angle between two eigenvectors is calculated using the cosine similarity method;
[0099] S23. Determine the similarity between the two feature vectors based on the cosine value of the angle between the two feature vectors.
[0100] Preferably, the formula for calculating the cosine value of the angle between two feature vectors using the cosine similarity method is:
[0101]
[0102] sim(v1,v2) represents the similarity between feature vectors V1 and V2;
[0103] V1 and V2 represent the feature vectors respectively;
[0104] || || represents the length of the feature vector;
[0105] · represents the dot product of eigenvectors.
[0106] It should be explained that the text input into the text encoder is converted into a feature vector representation, and then the similarity between the feature vectors is calculated to identify highly similar texts; the similarity between two text vectors is calculated using cosine similarity, which is an indicator that measures the degree of similarity between the directions of two vectors.
[0107] The theoretical basis of similarity calculation includes:
[0108] 1) Vector Space Model: In this model, text is represented as a vector and the semantic similarity of text can be calculated by operations between vectors.
[0109] 1) Cosine similarity: The similarity between two vectors is measured by calculating the cosine of the angle between them.
[0110] S3. Preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and determine whether the articles are similar.
[0111] Preferably, the preset similarity threshold is 0.8.
[0112] Preferably, a similarity threshold is preset, and the calculated cosine similarity is compared with the similarity threshold, and determining whether the documents are similar comprises the following steps:
[0113] If the calculated cosine similarity exceeds 0.8, the text is judged to be similar text;
[0114] If the calculated cosine similarity is lower than 0.8, the texts are judged to be dissimilar.
[0115] According to another embodiment of the present invention, Figure 2 As shown, a text duplicate checking system based on the LLM2Vec algorithm is also provided, and the text duplicate checking system includes: an LLM2Vec feature extraction module 1, a cosine similarity calculation module 2 and a similarity judgment module 3;
[0116] LLM2Vec feature extraction module 1, used to convert text into feature vectors using LLM2Vec algorithm;
[0117] The cosine similarity calculation module 2 is used to calculate the cosine similarity between each feature vector using the cosine similarity method;
[0118] The similarity judgment module 3 is used to preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and judge whether the documents are similar.
[0119] Preferably, the LLM2Vec feature extraction module 1 includes: a bidirectional model building module, a bidirectional model training module, a feature representation learning module and a text encoder module;
[0120] A bidirectional model building module for converting a decoder-only language model to a bidirectional model by replacing the causal attention mask with an all-ones matrix;
[0121] The bidirectional model training module is used to train the bidirectional model and optimize the weight of the bidirectional model;
[0122] The feature representation learning module is used to forward propagate the same input sequence twice through the optimized bidirectional model through the dropout mask to obtain two different representations of the same sequence and maximize the similarity of the two representations to obtain the feature representation;
[0123] The text encoder module is used to input the text into the text encoder and convert it into a feature vector using the text encoder.
[0124] In summary, with the aid of the above technical solution of the present invention, the present invention modifies the attention mechanism of the language model from unidirectional (considering only the previous token) to bidirectional (considering both the previous and next tokens). This enables the representation of each token to contain the information of the entire sequence, thereby generating a richer and more comprehensive text representation. The bidirectional attention mechanism helps to capture long-distance dependencies in the text, which is critical for understanding complex language structures and semantics. In unsupervised contrastive learning, the model generates two different representations of the same input sequence through a dropout mask, and then trains the model through a contrastive learning method so that the two representations are as similar as possible and as different as possible from the representations of other sequences. This method helps the model learn the ability to distinguish different texts, thereby generating high-quality text representations that can capture subtle differences between texts. In this way, the model can better distinguish between similar and dissimilar texts, which is very useful for tasks such as text clustering and similarity search. The present invention can not only capture the rich semantic information of the text, but also distinguish the subtle differences between different texts by generating a text encoder that represents high-quality text vectors, which is very valuable for various natural language processing tasks.
[0125] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are merely examples for the convenience of description and are not intended to limit the present invention. Those skilled in the art may make several changes and modifications without departing from the spirit and scope of the present invention. The scope of protection claimed by the present invention shall be based on the claims.
Claims
1. A text duplication checking method based on LLM2Vec algorithm, characterized in that: The text duplication checking method and system based on the LLM2Vec algorithm include the following steps: S1. Use LLM2Vec algorithm to convert text into feature vector; S2, using the cosine similarity method to calculate the cosine similarity between each feature vector; S3. Preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and determine whether the articles are similar.
2. A text duplication checking method based on LLM2Vec algorithm according to claim 1, characterized in that: The method of converting text into a feature vector using the LLM2Vec algorithm includes the following steps: S11, by replacing the causal attention mask with an all-one matrix and converting the decoder language-only model to a bidirectional model; S12, training the bidirectional model and optimizing the weight of the bidirectional model; S13, forward propagating the same input sequence twice through the optimized bidirectional model through the dropout mask, obtaining two different representations of the same sequence, and maximizing the similarity of the two representations to obtain the feature representation; S14. Input the text into a text encoder and convert it into a feature vector using the text encoder.
3. A text duplication checking method based on LLM2Vec algorithm according to claim 2, characterized in that: The method of replacing the causal attention mask with an all-one matrix and converting the decoder into a bidirectional model includes the following steps: S111, create a matrix with the same size as the attention matrix, and set all elements in the attention matrix to 1; S112. Based on the set attention matrix, in the attention layer of the decoder language model only, the causal attention mask is replaced with an all-1 matrix.
4. A text duplication checking method based on LLM2Vec algorithm according to claim 3, characterized in that: The method of forward-propagating the same input sequence twice through the optimized bidirectional model through the dropout mask to obtain two different representations of the same sequence and maximize the similarity of the two representations to obtain the feature representation includes the following steps: S131, generating a dropout mask for the input sequence; S132, using the dropout mask to perform two independent forward propagations on the input sequence, and in each propagation, discarding or retaining the corresponding neuron output according to the dropout mask; S133, two forward propagations obtain two different representations of the same input sequence; S134. Maximize the similarity between the two representations to obtain feature representation.
5. A text duplication checking method based on LLM2Vec algorithm according to claim 1, characterized in that: The method of calculating the cosine similarity between the feature vectors using the cosine similarity method comprises the following steps: S21, constructing a vector space model, and inputting two feature vectors into the vector space model; S22. In the vector space model, the cosine value of the angle between two eigenvectors is calculated using the cosine similarity method; S23. Determine the similarity between the two feature vectors based on the cosine value of the angle between the two feature vectors.
6. A text duplication checking method based on LLM2Vec algorithm according to claim 5, characterized in that: The formula for calculating the cosine value of the angle between two feature vectors using the cosine similarity method is: sim(v1,v2) represents the similarity between feature vectors V1 and V2; V1 and V2 represent the feature vectors respectively; || || represents the length of the feature vector; · represents the dot product of eigenvectors.
7. A text duplication checking method based on LLM2Vec algorithm according to claim 1, characterized in that: The preset similarity threshold is 0.
8.
8. A text duplication checking method based on LLM2Vec algorithm according to claim 7, characterized in that: The preset similarity threshold, comparing the calculated cosine similarity with the similarity threshold, and determining whether the documents are similar include the following steps: If the calculated cosine similarity exceeds 0.8, the text is judged to be similar text; If the calculated cosine similarity is lower than 0.8, the texts are judged to be dissimilar.
9. A text duplicate checking system based on the LLM2Vec algorithm, used to implement the text duplicate checking method based on the LLM2Vec algorithm described in any one of claims 1 to 8, characterized in that: The text duplication checking system includes: LLM2Vec feature extraction module, cosine similarity calculation module and similarity judgment module; LLM2Vec feature extraction module, used to convert text into feature vectors using the LLM2Vec algorithm; A cosine similarity calculation module is used to calculate the cosine similarity between each feature vector using a cosine similarity method; The similarity judgment module is used to preset a similarity threshold, compare the calculated cosine similarity with the similarity threshold, and judge whether the articles are similar.
10. A text duplication checking system based on LLM2Vec algorithm according to claim 9, characterized in that: The LLM2Vec feature extraction module includes: a bidirectional model construction module, a bidirectional model training module, a feature representation learning module and a text encoder module; A bidirectional model building module for converting a decoder-only language model to a bidirectional model by replacing the causal attention mask with an all-ones matrix; The bidirectional model training module is used to train the bidirectional model and optimize the weight of the bidirectional model; The feature representation learning module is used to forward propagate the same input sequence twice through the optimized bidirectional model through the dropout mask to obtain two different representations of the same sequence and maximize the similarity of the two representations to obtain the feature representation; The text encoder module is used to input the text into the text encoder and convert it into a feature vector using the text encoder.
Citation Information
Cited By
Text information compression method and compression device for index distributed database
CN121144271A
Text information compression method and compression device for indexing a distributed database
CN121144271B
