Text similarity detection method and device, storage medium and program product

Abstract

The invention relates to the field of computers, in particular to a text similarity detection method and device, a storage medium and a program product. The method comprises the steps of obtaining a target text; determining a first embedding vector corresponding to the target text; the first embedding vector is used for representing semantic features contained in the target text. According to the first embedded vector, n embedded vectors with the highest similarity with the first embedded vector in a vector database are determined, and n is a positive integer larger than 1; the vector database comprises embedding vectors corresponding to a plurality of preset texts respectively, and the n embedding vectors are embedding vectors corresponding to n preset texts respectively. Comparing semantics of the target text with semantics of the n preset texts by utilizing a large language model to obtain similarity information; the similarity information is used for indicating the semantic similarity between the target text and the n preset texts. The method is applied to text similarity detection.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method, device, storage medium and program product for text similarity detection. Background Art

[0002] At present, in order to check the originality of texts such as documents and materials, it is often necessary to check for plagiarism to determine whether the text to be checked has plagiarism, plagiarism or excessive citation. In order to check for plagiarism in the text to be checked, it is necessary to detect the similarity between the text to be checked and the existing text.

[0003] In the prior art, characters or words in the text to be detected are usually compared with characters or words in the existing text to determine the similarity between the text to be detected and the existing text. However, this method can only detect the literal similarity and cannot identify the semantic similarity between the texts. Therefore, this method has the problem of low accuracy.

[0004] Furthermore, how to efficiently and accurately detect the similarity between the text to be detected and the existing text is a technical problem that needs to be solved at present. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a method, device, storage medium and program product for text similarity detection.

[0006] In a first aspect, a method for text similarity detection is provided, the method comprising: obtaining a target text; determining a first embedding vector corresponding to the target text; the first embedding vector is used to represent semantic features contained in the target text; based on the first embedding vector, determining n embedding vectors in a vector database whose similarity with the first embedding vector exceeds a first similarity threshold, where n is a positive integer greater than 1; the vector database includes embedding vectors corresponding to a plurality of preset texts, and the n embedding vectors are embedding vectors corresponding to the n preset texts; using a large language model to compare the semantics of the target text with the n preset texts to obtain similarity information; the similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

[0007] On the one hand, in the above method of the embodiment of the present application, by comparing the first embedding vector with the embedding vectors of multiple preset texts in the vector database, n embedding vectors with a high similarity to the first embedding vector (i.e., the similarity exceeds the first similarity threshold) are determined. Among them, compared with the method of using a large language model to compare the semantic similarity between texts, the similarity between the embedding vectors is compared to screen out the preset texts (i.e., the above n preset texts) with a high similarity to the target text from a large number of preset texts while occupying less computing power resources. On the other hand, in the above method of the embodiment of the present application, after screening out the n preset texts with a high similarity to the target text, the large language model is also used to compare the semantics of the target text with the n preset texts and obtain similarity information. Among them, since the large language model can parse deeper semantics, the above similarity information obtained using the large language model can more accurately reflect the semantic similarity between the n texts and the target text. In summary, the above method provided by the embodiment of the present application can accurately determine the preset texts with a high similarity to the target text, and the semantic similarity between these preset texts and the target text while minimizing the computing power overhead.

[0008] In some implementations, determining the first embedding vector corresponding to the target text includes: using a text embedding model to determine the first embedding vector corresponding to the target text; the text embedding model is a text embedding model trained using a target sample set for text vectorization; the target sample set is a collection of text samples of the same content type as the target text.

[0009] In some implementations, the method further includes: determining the vector database according to the content type of the target text; the vector database includes embedding vectors corresponding to a plurality of preset texts corresponding to the content type.

[0010] In some implementations, determining the first embedding vector corresponding to the target text includes: preprocessing the target text to obtain an optimized text corresponding to the target text; the preprocessing includes: one or more of word segmentation, stop word removal, sentence cleaning, and text format unification; inputting the optimized text corresponding to the target text into a text embedding model for text vectorization to obtain a first embedding vector output by the text embedding model.

[0011] In some implementations, using a large language model to compare the semantics of the target text with the n preset texts to obtain similarity information includes: inputting the optimized text of the target text and the optimized texts corresponding to the n preset texts into the large language model, so that the large language model compares the semantics of the target text with the n preset texts based on the optimized text of the target text and the optimized texts corresponding to the n preset texts; and obtaining the similarity information output by the large language model.

[0012] In some implementations, the similarity information includes: one or more of: a similarity value between the target text and one or more preset texts, an identifier of the one or more preset texts, location information of the first target content in the one or more preset texts, and location information of the second target content in the target text; wherein the one or more preset texts are one or more preset texts among the n preset texts whose semantic similarity with the target text exceeds a semantic similarity threshold; the first target content includes content whose similarity with the content of the target text exceeds a second similarity threshold, and the second target content includes content whose similarity with the content in the one or more preset texts exceeds a third similarity threshold.

[0013] In a second aspect, a text similarity detection device is provided, comprising: an acquisition unit, configured to acquire a target text; a processing unit, configured to determine a first embedding vector corresponding to the target text; the first embedding vector is used to represent semantic features contained in the target text; the processing unit is further configured to determine, based on the first embedding vector, n embedding vectors in a vector database whose similarity to the first embedding vector exceeds a first similarity threshold, where n is a positive integer greater than 1; the vector database comprises embedding vectors corresponding to a plurality of preset texts, respectively, and the n embedding vectors are embedding vectors corresponding to the n preset texts, respectively; the processing unit is further configured to compare the semantics of the target text with the n preset texts using a large language model to obtain similarity information; the similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

[0014] In some implementations, the processing unit, used to determine the first embedding vector corresponding to the target text, includes: a processing unit, specifically used to determine the first embedding vector corresponding to the target text using a text embedding model; the text embedding model is a text embedding model for text vectorization trained using a target sample set; the target sample set is a collection of text samples of the same content type as the target text.

[0015] In some implementations, the processing unit is further used to determine the vector database according to the content type of the target text; the vector database includes embedding vectors corresponding to a plurality of preset texts corresponding to the content type.

[0016] In some implementations, the processing unit, used to determine the first embedding vector corresponding to the target text, includes: the processing unit, specifically used to preprocess the target text to obtain an optimized text corresponding to the target text; the preprocessing includes: one or more of word segmentation, stop word removal, sentence cleaning, and unified text format; the processing unit, specifically also used to input the optimized text corresponding to the target text into a text embedding model for text vectorization to obtain the first embedding vector output by the text embedding model.

[0017] In some implementations, the processing unit is further used to use a large language model to compare the semantics of the target text with the n preset texts to obtain similarity information, including: the processing unit is further used to input the optimized text of the target text and the optimized texts corresponding to the n preset texts into the large language model, so that the large language model compares the semantics of the target text with the n preset texts based on the optimized text of the target text and the optimized texts corresponding to the n preset texts; the processing unit is also used to obtain the similarity information output by the large language model.

[0018] In some implementations, the similarity information includes: one or more of: a similarity value between the target text and one or more preset texts, an identifier of the one or more preset texts, location information of the first target content in the one or more preset texts, and location information of the second target content in the target text; wherein the one or more preset texts are one or more preset texts among the n preset texts whose semantic similarity with the target text exceeds a semantic similarity threshold; the first target content includes content whose similarity with the content of the target text exceeds a second similarity threshold, and the second target content includes content whose similarity with the content in the one or more preset texts exceeds a third similarity threshold.

[0019] In a third aspect, a text similarity detection device is provided, comprising: a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to enable the text similarity detection device to implement the method as in the first aspect or any implementation manner of the first aspect when executing the computer program.

[0020] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the method as in the first aspect or any implementation manner in the first aspect.

[0021] According to a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to implement the method according to the first aspect or any implementation of the first aspect.

[0022] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0023] On the one hand, in the above method of the embodiment of the present application, by comparing the first embedding vector with the embedding vectors of multiple preset texts in the vector database, n embedding vectors with a high similarity to the first embedding vector (i.e., the similarity exceeds the first similarity threshold) are determined. Among them, compared with the method of using a large language model to compare the semantic similarity between texts, the similarity between the embedding vectors is compared to screen out the preset texts (i.e., the above n preset texts) with a high similarity to the target text from a large number of preset texts while occupying less computing power resources. On the other hand, in the above method of the embodiment of the present application, after screening out the n preset texts with a high similarity to the target text, the large language model is also used to compare the semantics of the target text with the n preset texts and obtain similarity information. Among them, since the large language model can parse deeper semantics, the above similarity information obtained using the large language model can more accurately reflect the semantic similarity between the n texts and the target text. In summary, the above method provided by the embodiment of the present application can accurately determine the preset texts with a high similarity to the target text, and the semantic similarity between these preset texts and the target text while minimizing the computing power overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 One of the flowcharts of a method for text similarity detection provided by an embodiment of the present disclosure;

[0027] Figure 2A second flowchart of a method for detecting text similarity provided in an embodiment of the present disclosure;

[0028] Figure 3 A third flowchart of a method for text similarity detection provided in an embodiment of the present disclosure;

[0029] Figure 4 A fourth flowchart of a method for detecting text similarity provided in an embodiment of the present disclosure;

[0030] Figure 5 One of the structural schematic diagrams of a text similarity detection device provided in an embodiment of the present disclosure;

[0031] Figure 6 The second structural diagram of a text similarity detection device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0034] The technical solution provided in the embodiments of the present application is introduced below.

[0035] In the embodiments of the present application, it is taken into consideration that: on the one hand, in order to ensure the accuracy of text similarity detection, it is necessary to identify the semantics of the text to be detected and the existing text and compare them; on the other hand, in order to ensure the accuracy of text similarity detection, it is also necessary to compare the text to be detected with as many existing texts as possible; on the other hand, in order to improve the efficiency of text similarity detection, it is also necessary to reduce the computing power overhead of the text similarity detection process as much as possible.

[0036] In view of the above situation, an embodiment of the present application provides a technical solution. In the technical solution, when performing similarity detection on the target text, after obtaining the target text, the target text can be first vectorized to obtain an embedding vector (hereinafter referred to as the first embedding vector) for representing the semantic features contained in the target text. In addition, the first embedding vector can be compared with the embedding vectors of multiple existing texts (hereinafter referred to as multiple preset texts) in the vector database, and then n embedding vectors whose similarity with the first embedding vector exceeds the first similarity threshold are determined from the embedding vectors of the multiple preset texts. In addition, a large language model (LLM) can be used to compare the semantics of the target text with n preset texts (i.e., the preset texts corresponding to the above n embedding vectors) and obtain similarity information, wherein the similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

[0037] It can be seen that, on the one hand, the above technical solution of the embodiment of the present application compares the first embedding vector with the embedding vectors of multiple preset texts in the vector database, thereby determining n embedding vectors with a high similarity to the first embedding vector (i.e., the similarity exceeds the first similarity threshold). Compared with the method of using a large language model to compare the semantic similarity between texts, comparing the similarity between embedding vectors can screen out preset texts with a high similarity to the target text (i.e., the above n preset texts) from a large number of preset texts while occupying less computing resources.

[0038] On the other hand, in the above technical solution of the embodiment of the present application, after screening out n preset texts with high similarity to the target text, the large language model is used to compare the semantics of the target text with the n preset texts and obtain similarity information. Since the large language model can parse deeper semantics, the above similarity information obtained by using the large language model can more accurately reflect the semantic similarity between the n texts and the target text.

[0039] In summary, the above technical solution provided by the embodiment of the present application can accurately determine the preset texts with a high similarity to the target text, as well as the semantic similarity between these preset texts and the target text, while minimizing the computing power overhead.

[0040] The technical solution provided by the embodiment of the present application is described in detail below with reference to examples. Specifically, the embodiment of the present application provides a method for text similarity detection. Among them, the execution subject of the method for text similarity detection can be a text similarity detection device. When the text similarity detection device is running, it can be used to execute all or part of the steps in the method for text similarity detection provided by the embodiment of the present application. Among them, in the actual application process, the function of the text similarity detection device can be implemented by a personal computer (including desktop computers, laptop computers, handheld computers and notebook computers, etc.), or smart phones, servers and other electronic devices; or, the function of the above-mentioned text similarity detection device can also be implemented by some hardware / software devices in the above-mentioned electronic devices. The embodiment of the present application does not impose any special restrictions on the specific form of the text similarity detection device.

[0041] like Figure 1 As shown, the text similarity detection method provided in the embodiment of the present application may include the following contents S101-S104:

[0042] S101. A text similarity detection device obtains a target text.

[0043] The target text may be any type of text data input into the text similarity detection device, for example, the target text may be text data of thesis, novel, script, etc.

[0044] S102: The text similarity detection device determines a first embedding vector corresponding to the target text.

[0045] The first embedding vector is used to represent the semantic features contained in the target text.

[0046] Specifically, the text similarity detection device may input the target text or information in the target text into a text embedding model for text vectorization to obtain a first embedding vector output by the text embedding model.

[0047] Among them, the text embedding model can specifically be a bidirectional encoder representation self-attention (BidirectionalEncoder Representations from Transformers, BERT) model, a robustly optimized BERT pretraining approach (RoBERTa) model, or a sentence embedding (Sentence Embeddings using Siamese BERT-Networks, Sentence-BERT) model using a twin BERT network. In addition, in actual application, the text similarity detection device can also use other types of text embedding models other than the above types to determine the first embedding vector corresponding to the target text. There is no restriction on the type of text embedding model in the embodiment of the present application.

[0048] In some implementations, in the embodiments of the present application, the above-mentioned text embedding model can specifically be a text embedding model for text vectorization trained using a target sample set.

[0049] The target sample set is a collection of text samples of the same content type as the target text. For example, if the target text is a script, the target sample set can be a collection of multiple existing scripts. For another example, if the target text is a communication paper, the target sample set can be a collection of multiple existing communication papers.

[0050] In this way, the text embedding model can be more adept at extracting features of texts of the target text type, thereby making the semantic features contained in the first embedding vector more accurate. For example, if the target text is a script, the text embedding model can be a model trained using multiple scripts (target sample sets) as training samples, or the text embedding model can be a model fine-tuned using multiple scripts (target sample sets) based on the trained model. In this case, the text embedding model is more adept at extracting features of script-type texts, thereby making the semantic features contained in the first embedding vector more accurate.

[0051] In addition, in some implementations, such as Figure 2 As shown, S102 may include:

[0052] S1021. The text similarity detection device preprocesses the target text to obtain an optimized text corresponding to the target text.

[0053] The preprocessing includes one or more of word segmentation, stop word removal, sentence cleaning, and text format unification.

[0054] Among them, in some designs, after the target text is preprocessed as above, the result obtained by the preprocessing can be used as the optimized text corresponding to the target text.

[0055] In some other designs, after the target text is preprocessed as described above, the preprocessing result may be further streamlined to obtain an optimized text corresponding to the target text.

[0056] The simplification operation may include extracting key information from the result obtained by the preprocessing. The optimized text corresponding to the target text specifically includes: key information corresponding to the target text.

[0057] Among them, the key information may include: one or more of the content type (such as papers, novels, scripts or essays, etc.), cited documents, outline summary or core content. For example, when the target text is a script, the key information of the above target text may include: a core plot overview, an outline summary and a main character story line (wherein the core plot overview may specifically include a one-sentence story. The outline summary may specifically include a high-level summary of the main line of the script. The main character story line may specifically be the behavior and development of the protagonist and important supporting roles, etc.). Exemplarily, the text similarity detection device may input the results obtained from the above preprocessing into a pre-trained natural language processing (NLP) model or a large language model, so that the NLP model or the large language model generates the above key information of the target text.

[0058] S1022. The text similarity detection device uses the optimized text corresponding to the target text as input into a text embedding model for text vectorization to obtain a first embedding vector output by the text embedding model.

[0059] The specific type and function of the text embedding model used in S1022 can refer to the above description. Specifically, the text embedding model can be a BERT model, a RoBERTa model, a Sentence-BERT model or other types of text embedding models. In addition, the text embedding model can be a text embedding model for text vectorization that is trained or fine-tuned using a target sample set.

[0060] In the above implementation, since the content of the optimized text is more concise than the original content of the target text, the extracted first embedding vector can more accurately reflect the characteristics of the target text.

[0061] S103. The text similarity detection device determines, based on the first embedding vector, n embedding vectors in a vector database whose similarity to the first embedding vector exceeds a first similarity threshold, where n is a positive integer greater than 1.

[0062] The vector database includes embedding vectors corresponding to a plurality of preset texts, and the n embedding vectors are embedding vectors corresponding to the n preset texts.

[0063] For example, the plurality of preset texts may specifically include: a plurality of existing documents and materials, etc. The vector database includes embedded vectors obtained by vectorizing the plurality of existing documents and materials.

[0064] For example, in actual application, Facebook AI Similarity Search (FAISS) or other similarity search engines can be used to find n embedding vectors with the highest similarity to the first embedding vector from the vector database.

[0065] Exemplarily, when searching for n embedded vectors whose similarity with the first embedded vector exceeds a first similarity threshold from a vector database, it is specifically possible to first determine the similarity between the first embedded vector and each embedded vector in the vector database by calculating the cosine similarity, and then search for n embedded vectors whose similarity with the first embedded vector exceeds the first similarity threshold.

[0066] For example, the following formula (1) can be used to calculate the first embedding vector u and each embedding vector v in the vector database respectively: i The similarity Sim(u,v i ):

[0067]

[0068] Among them, u represents the first embedding vector corresponding to the target text, v i represents the i-th embedded vector in the vector database (i≤k, k is the total number of embedded vectors in the vector database). Then, we get Sim(u,v i ) with the largest values ​​(i.e., the n embedding vectors with the highest similarity to the first embedding vector).

[0069] It should be noted that, in actual application, the first similarity threshold can be set according to actual application requirements. For example, when the cosine similarity is used to determine the similarity between the first embedded vector and each embedded vector in the vector database, the first similarity threshold can be a value between (0,1). In addition, when other methods are used to determine the similarity between the first embedded vector and each embedded vector in the vector database, the first similarity threshold can include a combination of one or more numerical values. For example, when judging the similarity between the first embedded vector and each embedded vector in the vector database from multiple dimensions, the first similarity threshold can be a combination of numerical values ​​including multiple dimensions, and then when the similarity between an embedded vector and multiple dimensions of the first embedded vector exceeds the numerical value of the corresponding dimension in the first similarity threshold, it is determined that the similarity between the embedded vector and the first embedded vector exceeds the first similarity threshold. There is no restriction on the specific form of the first similarity threshold in the embodiment of the present application.

[0070] In some implementations, such as Figure 3 As shown, before S103, the method may further include:

[0071] S105. The text similarity detection device determines a vector database according to the content type of the target text.

[0072] The vector database includes embedding vectors corresponding to a plurality of preset texts of the content type corresponding to the target text.

[0073] Specifically, multiple vector databases can be pre-constructed, wherein each vector database records the embedding vectors corresponding to texts of different content types. For example, multiple vector databases include: a vector database whose content type is a paper (i.e., the vector database records the embedding vectors corresponding to multiple paper texts), or a vector database whose content type is a script (i.e., the vector database records the embedding vectors corresponding to multiple script texts); for another example, multiple vector databases include: a vector database whose content type is suspense (i.e., the vector database records the embedding vectors corresponding to multiple suspense story texts), or a vector database whose content type is martial arts (i.e., the vector database records the embedding vectors corresponding to multiple martial arts story texts), etc.

[0074] Furthermore, a corresponding vector database may be determined from a plurality of vector databases according to the content type of the target text. For example, if the target text is a script, the vector database determined in S105 may include a plurality of existing scripts respectively corresponding to the embedding vectors.

[0075] Through the above implementation, the relevance between the preset text corresponding to each embedded vector in the vector database and the target text can be improved, so as to improve the accuracy of the result.

[0076] S104 , the text similarity detection device uses a large language model to compare the semantics of the target text with the n preset texts to obtain similarity information.

[0077] The n preset texts are the n preset texts corresponding to the n embedding vectors obtained in S103 whose similarity with the first embedding vector exceeds the first similarity threshold. The similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

[0078] Among them, the large language model used in the embodiments of the present application can be an artificial intelligence model with an architecture such as a generative pretrained transformer (GPT) for understanding and generating natural language text.

[0079] In some implementations, in a text similarity detection device, the target text and n preset texts can be input into a large language model, so that the large language model parses the semantics of the target text and the n preset texts and compares the semantics of the target text with the n preset texts, thereby obtaining the above-mentioned similarity information.

[0080] In some other implementations, S104 may specifically include:

[0081] S1041. The text similarity detection device inputs the optimized text of the target text and the optimized texts corresponding to the n preset texts into the large language model, so that the large language model compares the semantics of the target text and the n preset texts based on the optimized text of the target text and the optimized texts corresponding to the n preset texts.

[0082] S1042: The text similarity detection device obtains similarity information output by the large language model.

[0083] The content and generation process of the optimized text of the target text can refer to the corresponding S1021. The content and generation process of the optimized texts corresponding to the n preset texts can refer to the content and generation process of the optimized text of the target text.

[0084] In the above implementation, by inputting the optimized text of the target text and the optimized texts corresponding to the n preset texts into the large language model instead of inputting the entire contents of the target text and the n preset texts into the large language model, the amount of calculation of the large language model can be reduced and the efficiency of using the large language model can be improved.

[0085] In addition, in some designs, the above-mentioned similarity information may include: a similarity value between the target text and one or more preset texts.

[0086] Among them, the one or more preset texts are the one or more preset texts with the highest semantic similarity to the target text among the n preset texts. For example, the n preset texts include: text a, text b, text c, text d and text e. By comparing the semantics of the target text with the above texts a-text e through the large language model, it is determined that the semantic similarities with the target text are from large to small: text b, text c, text a, text e and text d. Then, according to the above sorting, the one or more preset texts with the highest similarity (i.e., one or more preset texts starting from text b) can be determined.

[0087] The form of the similarity value between the target text and one or more preset texts can be determined according to the actual application needs. For example, the similarity between the target text and each preset text can be expressed in percentage, and the similarity value between the target text and one or more preset texts can be a value in the interval [1,100]. For another example, the similarity between the target text and each preset text can be expressed in k preset gears, and the similarity value between the target text and one or more preset texts can be any item between 1 gear and k gear.

[0088] In the above design, by generating similarity information including similarity values ​​between the target text and one or more preset texts, the similarity information can clearly reflect the retrieved one or more preset texts that are most similar to the target text and the similarity values ​​between the target text and the one or more preset texts.

[0089] In addition, in some designs, the similarity information may include: one or more preset text identifiers.

[0090] Here, similar to the above design, the one or more preset texts are one or more preset texts having the highest semantic similarity with the target text among the n preset texts.

[0091] The identifier of one or more preset texts may specifically include information for uniquely identifying the preset texts. For example, the identifier of one or more preset texts may specifically be a combination of the title, book name, and author information of the preset texts.

[0092] In the above design, by generating similarity information including similarity values ​​between the target text and one or more preset texts, the similarity information can clearly reflect which preset texts are the one or more preset texts most similar to the target text.

[0093] In addition, in some designs, the above-mentioned similarity information may include: position information of the first target content in one or more preset texts.

[0094] Here, similar to the above design, the one or more preset texts are one or more preset texts having the highest semantic similarity with the target text among the n preset texts.

[0095] The first target content includes content whose similarity with the content of the target text exceeds the second similarity threshold. For example, the similarity information includes: some text paragraph content intercepted from one or more preset texts (i.e., the first target content) and the page numbers of these text paragraph contents in the one or more preset texts (i.e., the position information of the first target content in the one or more preset texts).

[0096] It should be noted that, in actual application, the second similarity threshold may be set according to actual application requirements, and the specific form of the second similarity threshold may not be limited in the embodiments of the present application.

[0097] In addition, in some designs, the above-mentioned similarity information may include: position information of the second target content in the target text.

[0098] Here, similar to the above design, the one or more preset texts are one or more preset texts having the highest semantic similarity with the target text among the n preset texts.

[0099] The second target content includes content whose similarity with the content in one or more preset texts exceeds a third similarity threshold. For example, the similarity information includes: some text paragraph content intercepted from the target text (i.e., the second target content) and the page numbers of these text paragraph contents in the target text (i.e., the position information of the second target content in the target text).

[0100] It is understandable that the above text introduces the content of similarity information through four designs respectively. In actual application, the content of similarity information can also be a combination of each or all of the above four designs. Exemplarily, the similarity information can specifically include all or part of the similarity value between the target text and one or more preset texts, the identifiers of one or more preset texts, the position information of the first target content in the one or more preset texts, and the position information of the second target content in the target text.

[0101] In addition, after obtaining the above similarity information, in the embodiment of the present application, the content in the similarity information can also be output in a preset form such as a picture, table or text for the user to consult. For example, the text similarity detection device can generate a picture, table or text including the above similarity information based on the above similarity information, and output it through a display screen or other device for the user to consult.

[0102] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present application also provides a text similarity detection device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the text similarity detection device in this embodiment can correspond to all the contents of the above method embodiment.

[0103] The present application provides a text similarity detection device. Figure 5 is a structural diagram of the text similarity detection device, such as Figure 5 As shown, the text similarity detection device 20 includes:

[0104] An acquisition unit 201 is used to acquire a target text;

[0105] The processing unit 202 is used to determine a first embedding vector corresponding to the target text; the first embedding vector is used to represent the semantic features contained in the target text;

[0106] The processing unit 202 is further configured to determine, based on the first embedding vector, n embedding vectors in a vector database whose similarity to the first embedding vector exceeds a first similarity threshold, where n is a positive integer greater than 1; the vector database includes embedding vectors corresponding to a plurality of preset texts, and the n embedding vectors are embedding vectors corresponding to the n preset texts;

[0107] The processing unit 202 is further used to compare the semantics of the target text with the n preset texts using a large language model to obtain similarity information; the similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

[0108] In some implementations, the processing unit 202, for determining a first embedding vector corresponding to the target text, includes:

[0109] Processing unit 202 is used to determine the first embedding vector corresponding to the target text using a text embedding model; the text embedding model is a text embedding model for text vectorization trained using a target sample set; the target sample set is a collection of text samples of the same content type as the target text.

[0110] In some implementations, the processing unit 202 is further configured to determine the vector database according to the content type of the target text; the vector database includes embedding vectors corresponding to a plurality of preset texts corresponding to the content type.

[0111] In some implementations, the processing unit 202, for determining a first embedding vector corresponding to the target text, includes:

[0112] A processing unit 202 is used to preprocess the target text to obtain an optimized text corresponding to the target text; the preprocessing includes: one or more of: word segmentation, stop word removal, sentence cleaning, and text format unification;

[0113] The processing unit 202 is further configured to input the optimized text corresponding to the target text into a text embedding model for text vectorization to obtain a first embedding vector output by the text embedding model.

[0114] In some implementations, the processing unit 202 is further configured to compare the semantics of the target text with the n preset texts using a large language model to obtain similarity information, including:

[0115] The processing unit 202 is further configured to input the optimized text of the target text and the optimized texts corresponding to the n preset texts into the large language model, so that the large language model compares the semantics of the target text with the n preset texts according to the optimized text of the target text and the optimized texts corresponding to the n preset texts;

[0116] The processing unit 202 is further configured to obtain similarity information output by the large language model.

[0117] In some implementations, the similarity information includes: one or more of: a similarity value between the target text and one or more preset texts, an identifier of the one or more preset texts, location information of the first target content in the one or more preset texts, and location information of the second target content in the target text; wherein the one or more preset texts are one or more preset texts among the n preset texts whose semantic similarity with the target text exceeds a semantic similarity threshold; the first target content includes content whose similarity with the content of the target text exceeds a second similarity threshold, and the second target content includes content whose similarity with the content in the one or more preset texts exceeds a third similarity threshold.

[0118] The text similarity detection device 20 provided in the embodiment of the present application can execute the text similarity detection method provided in any of the above embodiments, and its implementation principle and technical effect are similar, which will not be repeated here.

[0119] Based on the same inventive concept, an embodiment of the present application also provides a text similarity detection device. Figure 6 A schematic diagram of the structure of a text similarity detection device provided in an embodiment of the present application is shown in FIG. Figure 6As shown, the text similarity detection device provided in this embodiment includes: a memory 301 and a processor 302, wherein the memory 301 is used to store a computer program, and the processor 302 is used to execute the text similarity detection method provided in the above embodiment when executing the computer program.

[0120] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the method for text similarity detection provided in the above embodiment.

[0121] Based on the same inventive concept, an embodiment of the present application further provides a computer program product. When the computer program product is executed on a computer, the computing device implements the text similarity detection method provided in the above embodiment.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0123] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0124] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0125] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for text similarity detection, characterized in that: The method comprises: Get the target text; Determine a first embedding vector corresponding to the target text; the first embedding vector is used to represent the semantic features contained in the target text; According to the first embedding vector, n embedding vectors whose similarity with the first embedding vector exceeds a first similarity threshold are determined in a vector database, where n is a positive integer greater than 1; the vector database includes embedding vectors corresponding to a plurality of preset texts, and the n embedding vectors are embedding vectors corresponding to the n preset texts; The semantics of the target text and the n preset texts are compared by using a large language model to obtain similarity information; the similarity information is used to indicate the similarity of the semantics of the target text and the n preset texts.

2. The method according to claim 1, characterized in that The determining a first embedding vector corresponding to the target text includes: A first embedding vector corresponding to the target text is determined using a text embedding model; the text embedding model is a text embedding model trained using a target sample set for text vectorization; the target sample set is a collection of text samples of the same content type as the target text.

3. The method according to claim 1, characterized in that The method further comprises: The vector database is determined according to the content type of the target text; the vector database includes embedding vectors corresponding to a plurality of preset texts of the content type.

4. The method according to any one of claims 1 to 3, characterized in that: The determining a first embedding vector corresponding to the target text includes: Preprocessing the target text to obtain an optimized text corresponding to the target text; the preprocessing includes: one or more of word segmentation, stop word removal, sentence cleaning, and text format unification; The optimized text corresponding to the target text is input into a text embedding model for text vectorization to obtain a first embedding vector output by the text embedding model.

5. The method according to claim 4, characterized in that The using of the large language model to compare the semantics of the target text with the n preset texts to obtain similarity information includes: Inputting the optimized text of the target text and the optimized texts corresponding to the n preset texts into a large language model, so that the large language model compares the semantics of the target text and the n preset texts according to the optimized text of the target text and the optimized texts corresponding to the n preset texts; The similarity information output by the large language model is obtained.

6. The method according to any one of claims 1 to 3, characterized in that: The similarity information includes: one or more of: a similarity value between the target text and one or more preset texts, an identifier of the one or more preset texts, location information of the first target content in the one or more preset texts, and location information of the second target content in the target text; wherein the one or more preset texts are one or more preset texts among the n preset texts whose semantic similarity with the target text exceeds a semantic similarity threshold; the first target content includes content whose similarity with the content of the target text exceeds a second similarity threshold, and the second target content includes content whose similarity with the content in the one or more preset texts exceeds a third similarity threshold.

7. A text similarity detection device, characterized in that: include: An acquisition unit, used for acquiring a target text; A processing unit, configured to determine a first embedding vector corresponding to the target text; The first embedding vector is used to represent the semantic features contained in the target text; The processing unit is further used to determine, based on the first embedding vector, n embedding vectors in a vector database whose similarity with the first embedding vector exceeds a first similarity threshold, where n is a positive integer greater than 1; the vector database includes embedding vectors corresponding to a plurality of preset texts, and the n embedding vectors are respectively embedding vectors corresponding to the n preset texts; The processing unit is further used to compare the semantics of the target text with the n preset texts using a large language model to obtain similarity information; the similarity information is used to indicate the semantic similarity between the target text and the n preset texts.

8. A text similarity detection device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program, and the processor is used to enable the text similarity detection device to implement the method described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to implement the method according to any one of claims 1 to 6.

Images