Text similarity determination method and device, electronic equipment and storage medium

Abstract

The invention discloses a text similarity determination method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed text and at least one reference text associated with the to-be-processed text; for each reference text, determining a reference text in the to-be-processed text according to the first word distribution information of the to-be-processed text and the second word distribution information of the current reference text; removing the reference text from the to-be-processed text to obtain a to-be-duplicated text, and removing an associated text corresponding to the reference text from the current reference text to obtain a to-be-compared text; and based on at least one similarity algorithm, determining the text similarity of the to-be-duplicated text and the to-be-compared text. The problem that when the reference text exists in the text to be subjected to duplicate checking, the duplicate checking result of the text to be subjected to duplicate checking is inaccurate due to the reference text is solved, and the effect of obtaining the more accurate duplicate checking result is achieved by removing the reference text and conducting duplicate checking processing on the text after the reference text is removed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for determining text similarity. Background Art

[0002] At present, text duplication detection technology is widely used in scientific research institutions, journals and other institutions.

[0003] When checking for duplicate text, the text to be checked is usually compared one by one with the text in the text library to determine the degree of similarity in the text to be checked for duplicate text. However, in general, when the text to be checked for duplicate text includes quoted text, the quoted text in the text to be checked for duplicate text will seriously affect the checking result of the text to be checked for duplicate text, resulting in inaccurate text checking results.

[0004] In order to solve the above problems, it is necessary to improve the method of text duplication checking. Summary of the invention

[0005] The present invention provides a method, device, electronic device and storage medium for determining text similarity, so as to solve the problem that when there is a quoted text in the text to be checked for duplicates, the quoted text may lead to inaccurate duplicate checking results for the text to be checked for duplicates.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining text similarity, comprising:

[0007] Acquire a text to be processed and at least one reference text associated with the text to be processed;

[0008] For each reference text, determining a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text;

[0009] Eliminating the reference text from the text to be processed to obtain the text to be checked for duplicates, and eliminating the associated text corresponding to the reference text from the current reference text to obtain the text to be compared;

[0010] Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined.

[0011] In a second aspect, an embodiment of the present invention further provides a device for determining text similarity, comprising:

[0012] A text acquisition module, used to acquire a text to be processed and at least one reference text associated with the text to be processed;

[0013] A text determination module, for determining, for each reference text, a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text;

[0014] A text elimination module is used to eliminate the reference text from the text to be processed to obtain the text to be checked for duplicates, and to eliminate the associated text corresponding to the reference text from the current reference text to obtain the text to be compared;

[0015] The similarity determination module is used to determine the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining text similarity described in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining text similarity described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention is to obtain the text to be processed and at least one reference text associated with the text to be processed. In practical applications, after obtaining the text to be processed, at least one text is retrieved from the pre-built text library as a reference text for checking the text to be processed. Further, for each reference text, according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text, the quoted text in the text to be processed is determined. In this technical solution, in order to reduce the influence of the quoted content in the text to be processed on the result of checking the duplicate, after the text to be processed is segmented, at least one first text segmentation is obtained, and the word distribution information corresponding to each first text segmentation is determined according to the position distribution information of each first text segmentation in the text to be processed, so as to determine the corresponding word distribution information of the text to be processed according to the word distribution information of each first text segmentation. Similarly, the same text processing is performed on the current reference text to determine the second word distribution information corresponding to the current reference text. Further, based on the similarity between the first word distribution information and the second word distribution information, the quoted text is determined from the text to be processed, and the associated text corresponding to the quoted text in the current reference text is determined. The quoted text is removed from the text to be processed to obtain the text to be checked for duplicates, and the associated text corresponding to the quoted text is removed from the current reference text to obtain the text to be compared. Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined. The advantage of such a setting is that when the text to be checked for duplicates is checked based on the text to be compared, the influence of the quoted content on the duplicate checking result of the text to be processed can be effectively reduced, and a more accurate duplicate checking result can be obtained. The problem that when there is a quoted text in the text to be checked for duplicates, the duplicate checking result of the text to be checked for duplicates is inaccurate due to the quoted text is solved. By removing the quoted text and checking the text after removing the quoted text, a more accurate duplicate checking result can be achieved.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a method for determining text similarity according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a method for determining text similarity according to Embodiment 2 of the present invention;

[0026] Figure 3 is a flow chart of a method for determining text similarity according to Embodiment 2 of the present invention;

[0027] Figure 4 is a structural schematic diagram of a device for determining text similarity according to Embodiment 3 of the present invention;

[0028] Figure 5 The present invention is a schematic diagram of the structure of an electronic device for implementing the method for determining text similarity according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a method for determining text similarity is provided for the first embodiment of the present invention. This embodiment can be applied to obtaining a text to be processed and at least one reference text, and determining a quoted text in the text to be processed, and an associated text corresponding to the quoted text in the reference text, removing the quoted text in the text to be processed to obtain a text to be checked for duplicates, removing the associated text from the reference text to obtain a text to be compared, and performing a text check for duplicates on the text to be checked for duplicates based on the text to be compared to obtain a text similarity between the text to be checked for duplicates and the text to be compared, so as to characterize the similarity between the text to be compared and the text to be checked for duplicates based on the text similarity. The method can be executed by a device for determining text similarity, and the device for determining text similarity can be implemented in the form of hardware and / or software. The device for determining text similarity can be configured in a computing device that can execute a method for determining text similarity.

[0033] like Figure 1As shown, the method includes:

[0034] S110: Acquire a text to be processed and at least one reference text associated with the text to be processed.

[0035] The text to be processed can be understood as a text that needs to be checked for plagiarism and does not have any modification operations, and the modification operations include addition, deletion or modification operations. For example, the text to be processed can be a paper or journal that needs to be checked for plagiarism. The reference text can be understood as a text used to check for plagiarism on the text to be processed.

[0036] For example, a system that provides a paper duplication check function is used as an example. In the system, a text library for text duplication check is usually provided, and the text library includes a large number of published or uploaded paper texts or journal texts. After obtaining the text to be processed that needs to be checked for duplication, at least one paper text or journal text in the text library is retrieved as a reference text.

[0037] S120 . For each reference text, determine a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text.

[0038] In practical applications, before checking for duplicate content in the text to be processed, it is necessary to first perform word segmentation on the text to be processed to obtain at least one first text word segmentation. Taking one of the first text word segments as an example, the first word distribution information can be understood as the position distribution information of the first text word segmentation in the text to be processed, such as the first text word segmentation is located in the first paragraph, the body, and the last paragraph of the text in the text to be processed. Correspondingly, when checking for duplicate content in the text to be processed based on the reference text, the same operation needs to be performed on the reference text, that is, the word segmentation is prohibited for the reference text, at least one second text word segmentation is obtained, and the position distribution information of the second text word segmentation in the reference text is used as the second word distribution information. The reference text can be understood as the text content quoted from the reference text in the text to be processed. For example, the text content in the reference text is quoted in the text to be processed as an example, argumentation, or argument support, etc. At this time, the text content quoted from the reference text becomes the reference text. It is understandable that the quoted text in the text to be processed may be completely consistent with the quoted content in the reference text, or may have certain differences from the quoted content in the reference text, such as changes in noun names, changes in noun positions, and changes in the order of text sentences.

[0039] Specifically, in order to accurately check the text to be processed for duplicates, it is necessary to check the text to be processed for duplicates based on at least one reference text. For each reference text, taking one of the reference texts as the current reference text as an example, the text to be processed and the current reference text are respectively segmented to obtain at least one first text segmentation in the text to be processed and at least one second text segmentation in the current reference text. Furthermore, based on the distribution of each first text segmentation in the text to be processed, the first word distribution information corresponding to the text to be processed is determined, and based on the distribution of each second text segmentation in the current reference text, the second word distribution information corresponding to the current reference text is determined, so as to determine the referenced text in the text to be processed based on the first word distribution information and the second word distribution information.

[0040] It should be noted that the process of determining the first word distribution information corresponding to the text to be processed is similar to the process of determining the second word distribution information in the current reference text. Based on this, obtaining the first word distribution information is taken as an example for explanation.

[0041] Exemplarily, when checking for duplicates in a text to be processed, the text to be processed is first segmented to obtain at least one first text segmentation, such as the Jieba segmentation processing method. For each first text segmentation obtained by segmentation, the score vector of each first text segmentation in the text to be processed can be calculated based on the term frequency-inverse file frequency algorithm (i.e., TF-IDF), so as to measure the importance of the corresponding first text segmentation in the text to be processed according to the score vector corresponding to each first text segmentation. Among them, the higher the score of the first text segmentation, the higher the importance in the text to be processed. Furthermore, the first text segmentations are sorted according to their importance in the text to be processed, and the word distribution information corresponding to each first text segmentation is determined respectively.

[0042] Taking one of the first text segmentations as an example, the position information of the first text segmentation in the text to be processed is located to determine the first word distribution information of the first text segmentation in the text to be processed. In order to determine whether the content associated with the first text segmentation in the text to be processed is a quoted text, when the first text segmentation is included in the current reference text, it is used as the second text segmentation, and the second word distribution information of the second text segmentation in the current reference text is located. If the similarity between the first word distribution information and the second word distribution information is high, it can be determined that the text content associated with the first text segmentation is a quoted text quoted by the text to be processed from the current reference text.

[0043] S130, removing the quoted text from the text to be processed to obtain the text to be checked for duplicates, and removing the associated text corresponding to the quoted text from the current reference text to obtain the text to be compared.

[0044] The text to be checked for duplicates can be understood as the text to be processed after the reference text is removed. The associated text can be understood as the text content in the current reference text that corresponds to the reference text. The text to be compared can be understood as the current reference text after the associated text is removed.

[0045] In practical applications, the text information in other articles may be quoted in the writing process of the text to be processed to enrich the text content of the text to be processed. However, for the quoted text, the quoted text should be removed when checking the text to be processed to obtain the text to be checked for duplicates, and the duplicate checking is performed based on the text to be checked for duplicates to obtain a more accurate duplicate checking result. Correspondingly, when checking the text to be processed based on the current reference text, if the current reference text contains an associated text corresponding to the quoted text, the associated text should also be removed from the current reference text to obtain the text to be compared. Furthermore, the text to be checked for duplicates is checked based on the text to be compared.

[0046] The advantage of this setting is that the text to be processed inevitably needs to quote the content of other articles during the writing process. If the text to be processed is directly checked for plagiarism, the plagiarism rate will be too high when calculating the plagiarism check result of the text to be processed, that is, there is a problem of inaccurate plagiarism check result. In this solution, the text to be checked for plagiarism is obtained by removing the quoted text in the text to be processed, and the text to be checked for plagiarism is checked for plagiarism, which can effectively reduce the influence of the quoted text on the process of checking for plagiarism of the text to be processed, so that the plagiarism check result of the text to be processed is more accurate.

[0047] It should be noted that the so-called removal of the quoted text from the text to be processed and the removal of the associated text from the current reference text in the present technical solution are for the purpose of making it easier to understand the duplicate checking process of the present technical solution, rather than actually removing the quoted text and the associated text. For example, after determining the quoted text and the associated text, they can be specially marked so that when the text information carrying the special mark is detected during the text duplicate checking, the influence weight of the corresponding content on the text is set to the lowest, so as to reduce the influence of the quoted text on the duplicate checking of the text to be processed.

[0048] S140: Determine the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm.

[0049] Among them, text similarity can be understood as an indicator used to characterize the similarity between the text to be checked for duplicates and the text to be compared. The higher the text similarity, the higher the similarity between the text to be checked for duplicates and the text to be compared, and vice versa. The similarity algorithm can be a cosine similarity algorithm, a text similarity algorithm, a distance similarity algorithm, etc.

[0050] Optionally, based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined, including: vectorizing the text to be checked for duplicates and the text to be compared to obtain a first text vector corresponding to the text to be checked for duplicates, and a second text vector corresponding to the text to be compared; based on at least one similarity algorithm, determining the vector similarity between the first text vector and the second text vector, and using the vector similarity as the text similarity between the text to be checked for duplicates and the text to be compared.

[0051] Specifically, the text to be checked for duplicates is vectorized to obtain a first text vector corresponding to the text to be checked for duplicates. Wherein, the text to be checked for duplicates includes at least one text subsequence, and the first text vector includes a first subsequence vector corresponding to each text subsequence. Similarly, the text to be compared is vectorized to obtain a second text vector. Wherein, the text to be compared includes at least one text subsequence, and the second text vector includes a second subsequence vector corresponding to each text subsequence. Based on at least one similarity algorithm, the similarity of the first text vector and the second text vector is calculated to obtain vector similarity, and the vector similarity is used as the text similarity between the text to be checked for duplicates and the text to be compared, that is, the text similarity between the text to be processed and the current reference text.

[0052] The technical solution of the embodiment of the present invention is to obtain the text to be processed and at least one reference text associated with the text to be processed. In practical applications, after obtaining the text to be processed, at least one text is retrieved from the pre-built text library as a reference text for checking the text to be processed. Further, for each reference text, according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text, the quoted text in the text to be processed is determined. In this technical solution, in order to reduce the influence of the quoted content in the text to be processed on the result of checking the duplicate, after the text to be processed is segmented, at least one first text segmentation is obtained, and the word distribution information corresponding to each first text segmentation is determined according to the position distribution information of each first text segmentation in the text to be processed, so as to determine the corresponding word distribution information of the text to be processed according to the word distribution information of each first text segmentation. Similarly, the same text processing is performed on the current reference text to determine the second word distribution information corresponding to the current reference text. Further, based on the similarity between the first word distribution information and the second word distribution information, the quoted text is determined from the text to be processed, and the associated text corresponding to the quoted text in the current reference text is determined. The quoted text is removed from the text to be processed to obtain the text to be checked for duplicates, and the associated text corresponding to the quoted text is removed from the current reference text to obtain the text to be compared. Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined. The advantage of such a setting is that when the text to be checked for duplicates is checked based on the text to be compared, the influence of the quoted content on the duplicate checking result of the text to be processed can be effectively reduced, and a more accurate duplicate checking result can be obtained. The problem that when there is a quoted text in the text to be checked for duplicates, the duplicate checking result of the text to be checked for duplicates is inaccurate due to the quoted text is solved. By removing the quoted text and checking the text after removing the quoted text, a more accurate duplicate checking result can be achieved.

[0053] Embodiment 2

[0054] Figure 2 A flowchart of a method for determining text similarity provided in Embodiment 2 of the present invention, optionally, the first word distribution information of the text to be processed and the second word distribution information of the current reference text are refined.

[0055] like Figure 2 As shown, the method includes:

[0056] S210: Acquire a text to be processed and at least one reference text associated with the text to be processed.

[0057] S220, performing Jieba word segmentation processing on the target text to obtain at least one word segmentation to be screened.

[0058] The target text is the text to be processed or the current reference text.

[0059] It should be noted that when performing duplicate checking on the text to be processed based on the current reference text, similar operations need to be performed on the current reference text and the text to be processed. Based on this, in this embodiment, the target text is used as an example of the text to be processed or the current reference text, so as to more clearly understand this technical solution.

[0060] Among them, the candidate tokens to be screened can be understood as the tokens obtained after performing tokenization on the text to be processed. If the target text is the text to be processed, the candidate tokens to be screened can be understood as the first tokenized text in the text to be processed. If the target text is the current reference text, the candidate tokens to be screened can be understood as the second tokenized text in the current reference text.

[0061] In this embodiment, the target text is used as an example of the text to be processed for illustration.

[0062] Specifically, after obtaining the text to be processed, preprocess the text to be processed. For example, remove the stop words in the text to be processed, such as "de", "shi", "zai", etc. These words usually appear very frequently in the text but do not contain specific meanings and can be regarded as noise in the text. Therefore, such words need to be removed when preprocessing the text to be processed. Further, perform tokenization on the text to be processed based on Jieba Tokenizer to obtain at least one candidate token to be screened.

[0063] S230. Based on the term frequency–inverse document frequency (TF-IDF) algorithm, determine the score attributes corresponding to each candidate token to be screened, and determine at least one token to be used from at least one candidate token to be screened according to each score attribute.

[0064] Among them, the tokens to be used include high-frequency tokens or professional term tokens. In this technical solution, the higher the score attribute corresponding to the candidate token to be screened, the more capable it is of representing the text content of the text to be processed. Based on this, the candidate tokens to be screened with higher score attributes are used as the tokens to be used.

[0065] Among them, the term frequency–inverse document frequency (TF-IDF) algorithm is a weighting technique used for information retrieval and text mining, and can be used to evaluate the importance of a word for a document set or a single document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in the document, but at the same time decreases in inverse proportion to the frequency of its appearance in the corpus. In short, if a certain word appears more frequently in an article and rarely appears in other articles, it is considered that this word or phrase has good category discrimination ability, which means that the score attribute of this word or phrase in the article is higher.

[0066] Specifically, after obtaining at least one to-be-screened participle, in order to facilitate the calculation of the score attribute corresponding to each to-be-screened participle, each to-be-screened participle is vectorized, and the word vector corresponding to each to-be-screened participle is calculated based on the TF-IDF algorithm to perform score calculation, and the score attribute corresponding to each to-be-screened participle is obtained. It can be understood that the score attribute can be used to characterize the importance of the corresponding to-be-screened participle in the to-be-processed text. The higher the score attribute, the higher the importance of the corresponding to-be-screened participle in the to-be-processed text. Further, based on the score attribute of each to-be-screened participle, at least one to-be-used participle is obtained from at least one to-be-screened participle, such as sorting each to-be-screened participle according to the score attribute, and taking the top 50% of the to-be-screened participles as the to-be-used participles.

[0067] S240: Determine target word distribution information corresponding to the target text according to word distribution information of at least one word segment to be used.

[0068] The target word distribution information includes the first word distribution information corresponding to the text to be processed or the second word distribution information corresponding to the current reference text. In this embodiment, the target word distribution information refers to the first word distribution information in the text to be processed.

[0069] In practical applications, after obtaining at least one to-be-used segmented word, it is possible to determine whether there is a reference text in the to-be-processed text based on the distribution information of each to-be-used segmented word in the to-be-processed text. Specifically, in order to reduce the impact of the reference problem in the to-be-processed text on the duplicate checking result of the to-be-processed text, the word distribution information of each to-be-used segmented word is determined to determine the first word distribution information corresponding to the to-be-processed text.

[0070] Taking the word distribution information corresponding to one of the to-be-used participles as the first word distribution information as an example, optionally, according to the word distribution information of at least one to-be-used participle, determining the target word distribution information corresponding to the target text, including: for each to-be-used participle, determining the local text window corresponding to the current participle, and determining the nearest neighbor window corresponding to the local text window; performing linear normalization processing on the window distance between the local text window and the nearest neighbor window to obtain the word distribution information corresponding to the current participle; determining the target word distribution information corresponding to the target text according to the word distribution information corresponding to at least one to-be-used participle.

[0071] The local text window refers to the text window containing the current word segmentation, and the local text window has a certain window length, which can be determined according to the number of word segments that can be contained in the window. For example, if the window length is preset to 11, it means that the number of word segments that can be contained in the local text window corresponding to the current word segmentation is 11. The nearest neighbor window can be understood as the text window that contains the current word segmentation in the text to be processed and is closest to the window of the local text window.

[0072] In a specific example, Figure 3 As shown, the window length of the local text window corresponding to the current segmentation is preset to L=11, that is, the local distribution is measured by the 10 segmentations on both sides of the current segmentation, and in the process of text duplication checking, the similar parts between the texts are judged by the local text window corresponding to the current segmentation. For example, for the same quoted part in different texts, since the narrative order of the quoted underwear Rong may be different, the same surface description content in the text can be determined by the window similarity between the local text window and the nearest neighbor window, and the similarity of words can be measured when there is a specific field in the text.

[0073] Specifically, after obtaining the local text window corresponding to the current word segmentation, the nearest neighbor window corresponding to the local text window is determined. This includes: determining at least one window to be compared corresponding to the local text window; for each window to be compared, determining the window distance between the current window to be compared and the local window according to the difference in the score attribute between the word to be used contained in the current window to be compared and the current word segmentation; and determining the window to be compared corresponding to the smallest window to be compared distance in at least one window to be compared as the nearest neighbor window corresponding to the local text window.

[0074] The window to be compared can be understood as a text window containing the current segmentation other than the local text window corresponding to the current segmentation. The window distance to be compared can be understood as the window distance between the window to be compared and the local text window.

[0075] For example, if the current word segmentation is at the first paragraph of the text to be processed, the local text window containing the current word segmentation is taken as the local text window, and other text windows in the text to be processed that contain the current word segmentation except the local window are taken as windows to be compared. That is to say, in the text to be processed, the current word segmentation appears multiple times, and accordingly, the number of text windows containing the current word segmentation is also large. In order to distinguish the text windows, one of the text windows containing the current word segmentation is taken as the local text window, and the remaining text windows containing the current word segmentation are taken as windows to be compared. It should be noted that in addition to the current word segmentation, other word segmentations to be used may also be included in the local text window and the window to be compared.

[0076] Specifically, when determining the nearest neighbor window corresponding to the local text window, it can be determined in the following manner:

[0077] After obtaining the text window corresponding to each word segmentation to be used in the text to be processed, determine the nearest neighbor window corresponding to the local text window of the current word segmentation. Specifically, compare the window distances corresponding to each window to be compared (i.e., the text window corresponding to the word segmentation to be used) and the local distribution window in turn. For the word segmentation to be used between the window to be compared and the local text window, the distance between the same word segmentation to be used is 0. For the different word segmentations to be used between the window to be compared and the local text window, the corresponding relationship is determined according to the order of the windows from left to right, and the difference in the score attributes between the two word segmentations to be used is used as the window distance to be compared between the two windows. After determining the window distance to be compared between each window to be compared and the local text window, the window to be compared corresponding to the smallest window distance to be compared is determined as the nearest neighbor window.

[0078] Specifically, after obtaining the nearest neighbor window of the local text window corresponding to each to-be-used word segmentation, there is a to-be-compared window distance for each pair of nearest neighbor windows. For the local text window D of the i-th to-be-used word segmentation, i , and the local text window D of the nearest neighbor word j j The distance between ij :

[0079]

[0080] Among them, d ij Indicates the window distance between the local text window and the nearest neighbor window to be compared, TF-IDF iq Indicates the score attribute corresponding to the i-th word to be used, TF-IDF jq Represents the score attribute corresponding to the nearest neighbor word j.

[0081] On this basis, the distance between the windows to be compared between the nearest neighbor windows corresponding to the current word segmentation is linearly normalized to obtain the word distribution information corresponding to the current word segmentation:

[0082]

[0083] in, Indicates the word distribution information corresponding to the current word segmentation, d ij It represents the window distance between the local text window and the nearest neighbor window to be compared, and Norm represents the norm sign.

[0084] Based on the above operation, the position information of the local text window corresponding to the current word segmentation in the text to be processed can be determined, and then the word distribution information of the current word segmentation in the text to be processed can be determined. Based on this, according to the word distribution information of each word segmentation to be used, the first word distribution information corresponding to the text to be processed can be determined.

[0085] S250: Determine the reference text in the text to be processed.

[0086] In practical applications, in order to remove the quoted text in the text to be processed, it is necessary to accurately determine the quoted text in the text to be processed. Optionally, determining the quoted text in the text to be processed includes: determining at least one nearest neighbor curve associated with the local text window; performing text segmentation on the target text according to the number of at least one nearest neighbor curve to obtain at least one text subsequence; and determining the quoted text in the text to be processed based on the at least one text subsequence.

[0087] The target text is the text to be processed. The text subsequence can be understood as a subsequence obtained by dividing the text to be processed into words, phrases or paragraphs.

[0088] In practical applications, the number of nearest neighbor curves corresponding to the corresponding word segmentation to be used can be corrected through the window distribution information corresponding to the text window corresponding to each word segmentation to be used, and the text to be processed can be divided according to the number of nearest neighbor curves.

[0089] For details, see again Figure 3 , after obtaining the nearest neighbor relationship of the local text window corresponding to each to-be-used segmentation word, the text can be divided through the nearest neighbor curve between the local text windows. In the corresponding relationship between the nearest neighbor windows, there are several corresponding nearest neighbor relationships above each to-be-used segmentation word, that is, the number of nearest neighbor relationships between the local text windows on both sides of the local text window of each to-be-used segmentation word. The number of nearest neighbor relationships on both sides can be used as the text segmentation standard for the text to be processed. It can be understood that the more nearest neighbor relationships there are on both sides of the local text window of a to-be-used segmentation word, the more similar the to-be-used segmentation word is, and the fewer the number, the more there is no nearest neighbor relationship on both sides of the to-be-used segmentation word, that is, there is almost no correlation between the content on both sides of the to-be-used segmentation word. Then the corresponding number of nearest neighbor windows on both sides of each to-be-used segmentation word can be used to measure the segmentation degree of each to-be-used segmentation word. For the number of nearest neighbor relationships N on both sides of each to-be-used segmentation word i , linearly normalize the number of each word to be used to obtain the degree of separability of each word to be used ξ i For the to-be-used segmented words with a segmentation degree higher than 0.5, all segmentation results are obtained as segmentation points, that is, at least one text subsequence is obtained.

[0090] Optionally, based on at least one text subsequence, determining the quoted text in the text to be processed includes: for each text subsequence, determining a score attribute of the current subsequence in at least one local text window corresponding to the word segmentation to be used, and determining an average score attribute corresponding to the current subsequence; if the average score attribute is less than a preset score attribute, determining that the current subsequence is a quoted text.

[0091] After obtaining all the segmentation results, for each segmented text subsequence, the average score of the entire text subsequence is obtained by the average TF-IDF score corresponding to the to-be-used segmentation words in each local text window (i.e., the average score attribute corresponding to multiple to-be-used segmentation words), and then the text subsequences with low importance are determined by the average score attribute of each text subsequence and removed. The average score attribute of the mth text subsequence in all text subsequences is normalized by the scores of all text subsequences to obtain the importance of each text subsequence ε m :

[0092]

[0093] Among them, TF-IDF wn represents the score attribute of the nth word to be used in the wth local text window in the mth text subsequence, and Norm represents the norm symbol.

[0094] In practical applications, at least one text subsequence is obtained after the text to be processed is divided, and each text subsequence includes at least one local text window corresponding to the word segment to be used. Further, taking one of the text subsequences as the current subsequence as an example, the number of local text windows in the current subsequence is determined, and the corresponding score attribute of the corresponding local text window is determined according to the score attribute corresponding to the word segment to be used in each local text window. Further, the average score attribute corresponding to all local text windows in the current subsequence is determined, and if the average score attribute is less than the preset score attribute, the current subsequence is determined to be a reference text.

[0095] The advantage of this setting is that by determining the number of nearest neighbor relationships of text windows on both sides of each local text window to divide the text to be processed, the text to be processed can be divided into different text subsequences. In the subsequent comparison process with the text information in the text library, the similarity is determined in the reference text according to each text subsequence, and this similarity is used as an optimization factor in the cosine similarity measurement process. The higher the similarity, the more important the local window of the divided text subsequence is, so that in the subsequent process, the text similarity between the two texts can be determined based on the text similarity between the text to be processed and the reference text.

[0096] It should be noted that in actual operation, the text processing operation on the text to be processed is similar to the text processing operation corresponding to the current reference text. Based on this, the reference text corresponding to the text to be processed can be obtained, and the associated text corresponding to the reference text in the current reference text can be determined.

[0097] S260, removing the quoted text from the text to be processed to obtain the text to be checked for duplicates, and removing the associated text corresponding to the quoted text from the current reference text to obtain the text to be compared.

[0098] S270: Determine the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm.

[0099] Based on the above example, each text subsequence contains one or more local text windows corresponding to the to-be-used segmented words. When measuring the importance of each text subsequence in the current reference text, the more text subsequences exist in the current reference text, the higher the optimization factor for the to-be-used segmented words contained in the text subsequence in the cosine similarity measurement.

[0100] For example, for the mth text subsequence in the text to be checked for duplicate content, the optimization factor can be expressed by the following formula:

[0101]

[0102] Among them, N′ md Indicates the number of text subsequences in the current reference text whose similarity to the mth text subsequence is higher than the preset similarity, N d Indicates the number of subsequences in the current reference text that include the length of the m-th text subsequence.

[0103] After obtaining the cosine similarity optimization factor, the cosine similarity of the feature vectors of the text to be checked for duplicates and the text to be compared is obtained by multiplying the score attribute of each dimension (i.e., the participle to be used) and η, and then calculating the angle of the feature vectors to obtain the accurate cosine similarity. The obtained cosine similarity is the text similarity between the text to be checked for duplicates and the current reference text, that is, the text similarity between the text to be processed and the current reference text.

[0104] The technical solution of the embodiment of the present invention is to obtain the text to be processed and at least one reference text associated with the text to be processed. In practical applications, after obtaining the text to be processed, at least one text is retrieved from the pre-built text library as a reference text for checking the text to be processed. Further, for each reference text, according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text, the quoted text in the text to be processed is determined. In this technical solution, in order to reduce the influence of the quoted content in the text to be processed on the result of checking the duplicate, after the text to be processed is segmented, at least one first text segmentation is obtained, and the word distribution information corresponding to each first text segmentation is determined according to the position distribution information of each first text segmentation in the text to be processed, so as to determine the corresponding word distribution information of the text to be processed according to the word distribution information of each first text segmentation. Similarly, the same text processing is performed on the current reference text to determine the second word distribution information corresponding to the current reference text. Further, based on the similarity between the first word distribution information and the second word distribution information, the quoted text is determined from the text to be processed, and the associated text corresponding to the quoted text in the current reference text is determined. The quoted text is removed from the text to be processed to obtain the text to be checked for duplicates, and the associated text corresponding to the quoted text is removed from the current reference text to obtain the text to be compared. Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined. The advantage of such a setting is that when the text to be checked for duplicates is checked based on the text to be compared, the influence of the quoted content on the duplicate checking result of the text to be processed can be effectively reduced, and a more accurate duplicate checking result can be obtained. The problem that when there is a quoted text in the text to be checked for duplicates, the duplicate checking result of the text to be checked for duplicates is inaccurate due to the quoted text is solved. By removing the quoted text and checking the text after removing the quoted text, a more accurate duplicate checking result can be achieved.

[0105] Embodiment 3

[0106] Figure 4 This is a schematic diagram of a structure of a device for determining text similarity provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a text acquisition module 310 , a text determination module 320 , a text removal module 330 and a similarity determination module 340 .

[0107] The text acquisition module 310 is used to acquire the text to be processed and at least one reference text associated with the text to be processed;

[0108] A text determination module 320, for determining, for each reference text, a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text;

[0109] The text elimination module 330 is used to eliminate the reference text from the text to be processed to obtain the text to be checked for duplicates, and to eliminate the associated text corresponding to the reference text from the current reference text to obtain the text to be compared;

[0110] The similarity determination module 340 is used to determine the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm.

[0111] The technical solution of the embodiment of the present invention is to obtain the text to be processed and at least one reference text associated with the text to be processed. In practical applications, after obtaining the text to be processed, at least one text is retrieved from the pre-built text library as a reference text for checking the text to be processed. Further, for each reference text, according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text, the quoted text in the text to be processed is determined. In this technical solution, in order to reduce the influence of the quoted content in the text to be processed on the result of checking the duplicate, after the text to be processed is segmented, at least one first text segmentation is obtained, and the word distribution information corresponding to each first text segmentation is determined according to the position distribution information of each first text segmentation in the text to be processed, so as to determine the corresponding word distribution information of the text to be processed according to the word distribution information of each first text segmentation. Similarly, the same text processing is performed on the current reference text to determine the second word distribution information corresponding to the current reference text. Further, based on the similarity between the first word distribution information and the second word distribution information, the quoted text is determined from the text to be processed, and the associated text corresponding to the quoted text in the current reference text is determined. The quoted text is removed from the text to be processed to obtain the text to be checked for duplicates, and the associated text corresponding to the quoted text is removed from the current reference text to obtain the text to be compared. Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined. The advantage of such a setting is that when the text to be checked for duplicates is checked based on the text to be compared, the influence of the quoted content on the duplicate checking result of the text to be processed can be effectively reduced, and a more accurate duplicate checking result can be obtained. The problem that when there is a quoted text in the text to be checked for duplicates, the duplicate checking result of the text to be checked for duplicates is inaccurate due to the quoted text is solved. By removing the quoted text and checking the text after removing the quoted text, a more accurate duplicate checking result can be achieved.

[0112] Optionally, the text acquisition module includes: a to-be-screened word segmentation determination submodule, which is used to perform a stammering word segmentation process on the target text to obtain at least one to-be-screened word segmentation; wherein the target text is the to-be-processed text or the current reference text;

[0113] The submodule for determining the segmented words to be used is used to determine the score attributes corresponding to each segmented word to be screened based on the word frequency-reverse document frequency algorithm, and determine at least one segmented word to be used from at least one segmented word to be screened according to each score attribute; wherein the segmented words to be used include high-frequency segmented words or professional term segmented words;

[0114] The word distribution information determination submodule is used to determine the target word distribution information corresponding to the target text based on the word distribution information of at least one word segment to be used; wherein the target word distribution information includes the first word distribution information corresponding to the text to be processed or the second word distribution information corresponding to the current reference text.

[0115] Optionally, the word distribution information determination submodule includes: a nearest neighbor window determination unit, which is used to determine, for each word segment to be used, a local text window corresponding to the current word segment, and determine a nearest neighbor window corresponding to the local text window;

[0116] A normalization processing unit, used for performing linear normalization processing on the window distance between the local text window and the nearest neighbor window to obtain word distribution information corresponding to the current word segmentation;

[0117] The word distribution information determining unit is used to determine the target word distribution information corresponding to the target text according to the word distribution information corresponding to at least one to-be-used word segment.

[0118] Optionally, the nearest neighbor window determination unit includes: a to-be-compared window determination subunit, configured to determine at least one to-be-compared window corresponding to the local text window;

[0119] The window distance determination subunit is used to determine the window distance between the current window to be compared and the local window according to the difference in score attributes between the to-be-used segmentation contained in the current window to be compared and the current segmentation for each window to be compared;

[0120] The nearest neighbor window determination subunit is used to determine the window to be compared corresponding to the smallest distance from the window to be compared in at least one window to be compared as the nearest neighbor window corresponding to the local text window.

[0121] Optionally, the text determination module includes: a nearest neighbor curve determination submodule, configured to determine at least one nearest neighbor curve associated with the local text window;

[0122] A text subsequence determination submodule, used for performing text segmentation on the target text according to the number of at least one nearest neighbor curve to obtain at least one text subsequence;

[0123] The reference text determination submodule is used to determine the reference text in the text to be processed based on at least one text subsequence.

[0124] Optionally, the reference text determination submodule includes: a score attribute determination unit, which is used to determine, for each text subsequence, a score attribute of the current subsequence in at least one local text window corresponding to a to-be-used word segmentation, and determine an average score attribute corresponding to the current subsequence;

[0125] The reference text determination unit is used to determine that the current subsequence is a reference text if the average score attribute is less than a preset score attribute.

[0126] Optionally, the similarity determination module includes: a text vector determination submodule, which is used to perform vectorization processing on the text to be checked for duplicates and the text to be compared, to obtain a first text vector corresponding to the text to be checked for duplicates, and a second text vector corresponding to the text to be compared;

[0127] The similarity determination submodule is used to determine the vector similarity between the first text vector and the second text vector based on at least one similarity algorithm, and use the vector similarity as the text similarity between the text to be checked for duplicates and the text to be compared.

[0128] The apparatus for determining text similarity provided in the embodiment of the present invention can execute the method for determining text similarity provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Embodiment 4

[0130] Figure 5 A schematic diagram of the structure of an electronic device 10 of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0131] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for determining text similarity.

[0134] In some embodiments, the method for determining text similarity can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for determining text similarity described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the method for determining text similarity in any other appropriate manner (for example, by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The computer program for implementing the method for determining text similarity of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0140] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0142] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining text similarity, characterized in that: include: Acquire a text to be processed and at least one reference text associated with the text to be processed; For each reference text, determining a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text; Eliminating the reference text from the text to be processed to obtain the text to be checked for duplicates, and eliminating the associated text corresponding to the reference text from the current reference text to obtain the text to be compared; Based on at least one similarity algorithm, the text similarity between the text to be checked for duplicates and the text to be compared is determined.

2. The method according to claim 1, characterized in that The first word distribution information based on the to-be-processed text and the second word distribution information based on the current reference text includes: Performing Jieba word segmentation processing on the target text to obtain at least one word segmentation to be screened; wherein the target text is the text to be processed or the current reference text; Based on the word frequency-inverse document frequency algorithm, the score attribute corresponding to each to-be-screened segmentation word is determined, and at least one to-be-used segmentation word is determined from at least one of the to-be-screened segmentations according to each score attribute; wherein the to-be-used segmentation word includes a high-frequency segmentation word or a professional term segmentation word; According to the word distribution information of at least one of the word segments to be used, target word distribution information corresponding to the target text is determined; wherein the target word distribution information includes first word distribution information corresponding to the text to be processed or second word distribution information corresponding to the current reference text.

3. The method according to claim 2, characterized in that The determining target word distribution information corresponding to the target text according to the word distribution information of at least one of the to-be-used segmented words includes: For each word segment to be used, determine a local text window corresponding to the current word segment, and determine a nearest neighbor window corresponding to the local text window; Performing linear normalization processing on the window distance between the local text window and the nearest neighbor window to obtain word distribution information corresponding to the current word segmentation; According to the word distribution information corresponding to at least one of the to-be-used word segments, target word distribution information corresponding to the target text is determined.

4. The method according to claim 3, characterized in that The determining of the nearest neighbor window corresponding to the local text window comprises: Determine at least one window to be compared corresponding to the local text window; For each window to be compared, a window distance between the current window to be compared and the local window is determined according to a difference in score attributes between the to-be-used word contained in the current window to be compared and the current word; Among at least one of the windows to be compared, the window to be compared that has the smallest distance to the window to be compared is determined as the nearest neighbor window to the local text window.

5. The method according to claim 4, characterized in that The step of determining the reference text in the text to be processed includes: determining at least one nearest neighbor curve associated with the local text window; Performing text segmentation on the target text according to the number of at least one nearest neighbor curve to obtain at least one text subsequence; Based on at least one of the text subsequences, a reference text in the text to be processed is determined.

6. The method according to claim 5, characterized in that The step of determining the reference text in the to-be-processed text based on at least one of the text subsequences includes: For each text subsequence, determine a score attribute of the current subsequence in at least one local text window corresponding to a to-be-used word segmentation, and determine an average score attribute corresponding to the current subsequence; If the average score attribute is less than a preset score attribute, the current subsequence is determined to be a quoted text.

7. The method according to claim 1, characterized in that The determining of the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm includes: Performing vectorization processing on the text to be checked for duplicates and the text to be compared to obtain a first text vector corresponding to the text to be checked for duplicates and a second text vector corresponding to the text to be compared; Based on at least one similarity algorithm, the vector similarity between the first text vector and the second text vector is determined, and the vector similarity is used as the text similarity between the text to be checked for duplicates and the text to be compared.

8. A device for determining text similarity, characterized in that: include: A text acquisition module, used to acquire a text to be processed and at least one reference text associated with the text to be processed; A text determination module, for determining, for each reference text, a reference text in the text to be processed according to the first word distribution information of the text to be processed and the second word distribution information of the current reference text; A text elimination module is used to eliminate the reference text from the text to be processed to obtain the text to be checked for duplicates, and to eliminate the associated text corresponding to the reference text from the current reference text to obtain the text to be compared; The similarity determination module is used to determine the text similarity between the text to be checked for duplicates and the text to be compared based on at least one similarity algorithm.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining text similarity according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining text similarity according to any one of claims 1 to 7 when executed.

Images