Text similarity calculation method and related device

Abstract

The invention discloses a text similarity calculation method and a related device, and relates to the field of text duplicate checking, and the text similarity calculation method comprises the following steps: obtaining a first text and a second text to be subjected to similarity calculation; the method comprises the following steps: firstly, respectively carrying out diversified sequence extraction on a first text and a second text to obtain at least one group of sequences; respectively calculating the similarity of each group of sequences; and finally, carrying out weighted summation on the similarity of each group of sequences, and calculating to obtain the similarity of the first text and the second text. According to the scheme, the similarity calculation is performed on the diversified information in the text, and the final similarity is obtained by fusing the similarities of the diversified information, so that the similarity calculation accuracy of the diversified text can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of text duplication checking, and in particular to a text similarity calculation method and related devices. Background Art

[0002] With the rapid development of informatization, text duplication detection technology has been widely used in education, publishing, scientific research and other fields. With the development of technology in various fields, texts in various fields have become more and more diversified. Texts containing multiple languages ​​​​(such as texts containing characters in multiple languages ​​​​) and texts containing different character types (such as symbols, formulas, etc.) (such as science test questions containing complex symbols and formulas) have emerged. The demand for duplicate detection of diversified texts is becoming more urgent.

[0003] At present, text similarity calculation methods are mostly used to check for duplicate texts. However, existing text similarity calculation methods are unable to adapt to the similarity calculation of texts containing multiple languages ​​(such as texts containing characters in multiple languages) and texts containing different character types (such as symbols, formulas, etc.) (such as science test questions containing complex symbols and formulas). This results in poor accuracy in similarity calculation of diversified texts, which in turn leads to deviations in the duplicate checking results of diversified texts.

[0004] Therefore, how to provide a text similarity calculation method to improve the accuracy of similarity calculation of diversified texts has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of the above problems, the present application provides a text similarity calculation method and related devices to achieve the purpose of improving the accuracy of similarity calculation of diversified texts. The specific scheme is as follows:

[0006] The first aspect of the present application provides a method for calculating text similarity, comprising:

[0007] Obtain a first text and a second text to be subjected to similarity calculation;

[0008] Performing diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences;

[0009] Calculating the similarity of the sequences in each group respectively;

[0010] Based on the similarities of each group of sequences, the similarity between the first text and the second text is calculated.

[0011] In a possible implementation, obtaining the first text and the second text to be subjected to similarity calculation includes:

[0012] Obtaining a first original text and a second original text to be subjected to similarity calculation;

[0013] The first original text and the second original text are preprocessed to obtain a preprocessed first text and a preprocessed second text, wherein the preprocessed first text and the preprocessed second text are the first text and the second text to be subjected to similarity calculation.

[0014] In a possible implementation, performing multi-sequence extraction on the first text and the second text respectively to obtain at least one set of sequences includes:

[0015] Performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence;

[0016] And, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence.

[0017] In a possible implementation, respectively calculating the similarity of each group of sequences includes:

[0018] Calculating a plain text sequence similarity based on the first plain text sequence and the second plain text sequence;

[0019] Symbol sequence similarity is calculated based on the first symbol sequence and the second symbol sequence.

[0020] In a possible implementation, the extracting multivariate sequences from the first text and the second text respectively to obtain at least one set of sequences further includes:

[0021] Extracting a formula sequence from the first text to obtain a first formula sequence;

[0022] And, extracting a formula sequence from the second text to obtain a second formula sequence.

[0023] In a possible implementation, respectively calculating the similarity of each group of sequences further includes:

[0024] The formula sequence similarity is calculated based on the first formula sequence and the second formula sequence.

[0025] In a possible implementation, the calculating the similarity between the first text and the second text based on the similarities of each group of sequences includes:

[0026] Determining a weight for plain text sequence similarity and a weight for symbol sequence similarity;

[0027] Based on the weight of the plain text sequence similarity and the weight of the symbol sequence similarity, a weighted sum is performed on the plain text sequence similarity and the symbol sequence similarity to obtain the similarity between the first text and the second text.

[0028] In a possible implementation, the calculating the similarity between the first text and the second text based on the similarities of each group of sequences includes:

[0029] Obtaining a preset weight of symbol sequence similarity and a preset weight of non-symbol sequence similarity;

[0030] Calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity;

[0031] Based on the weight of the symbol sequence similarity, the weight of the non-symbol sequence similarity, the weight of the formula sequence similarity, and the weight of the plain text sequence similarity, the formula sequence similarity, the plain text sequence similarity, and the symbol sequence similarity are weighted and summed to obtain the similarity between the first text and the second text.

[0032] In a possible implementation, the calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity includes:

[0033] Calculating the weight of the similarity of the formula sequence based on the proportion of the first formula sequence and the second formula sequence in the non-symbol sequence;

[0034] The weight of the plain text sequence similarity is calculated based on the proportion of the first plain text sequence and the second plain text sequence in the non-symbol sequence.

[0035] A second aspect of the present application provides a text similarity calculation device, comprising:

[0036] An acquisition unit, used for acquiring a first text and a second text to be subjected to similarity calculation;

[0037] A sequence extraction unit, used for performing diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences;

[0038] A sequence similarity calculation unit, used to calculate the similarity of each group of sequences respectively;

[0039] The text similarity calculation unit is used to calculate the similarity between the first text and the second text based on the similarity of each group of sequences.

[0040] In a possible implementation, the acquiring unit is specifically configured to:

[0041] Obtaining a first original text and a second original text to be subjected to similarity calculation;

[0042] The first original text and the second original text are preprocessed to obtain a preprocessed first text and a preprocessed second text, wherein the preprocessed first text and the preprocessed second text are the first text and the second text to be subjected to similarity calculation.

[0043] In a possible implementation, the sequence extraction unit is specifically used to:

[0044] Performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence;

[0045] And, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence.

[0046] In a possible implementation, the sequence similarity calculation unit is specifically used to:

[0047] Calculating a plain text sequence similarity based on the first plain text sequence and the second plain text sequence;

[0048] Symbol sequence similarity is calculated based on the first symbol sequence and the second symbol sequence.

[0049] In a possible implementation, the sequence extraction unit is further specifically configured to:

[0050] Extracting a formula sequence from the first text to obtain a first formula sequence;

[0051] And, extracting a formula sequence from the second text to obtain a second formula sequence.

[0052] In a possible implementation, the sequence similarity calculation unit is further specifically used for:

[0053] The formula sequence similarity is calculated based on the first formula sequence and the second formula sequence.

[0054] In a possible implementation, the text similarity calculation unit is specifically used to:

[0055] Determining a weight for plain text sequence similarity and a weight for symbol sequence similarity;

[0056] Based on the weight of the plain text sequence similarity and the weight of the symbol sequence similarity, a weighted sum is performed on the plain text sequence similarity and the symbol sequence similarity to obtain the similarity between the first text and the second text.

[0057] In a possible implementation, the text similarity calculation unit is specifically used to:

[0058] Obtaining a preset weight of symbol sequence similarity and a preset weight of non-symbol sequence similarity;

[0059] Calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity;

[0060] Based on the weight of the symbol sequence similarity, the weight of the non-symbol sequence similarity, the weight of the formula sequence similarity, and the weight of the plain text sequence similarity, the formula sequence similarity, the plain text sequence similarity, and the symbol sequence similarity are weighted and summed to obtain the similarity between the first text and the second text.

[0061] In a possible implementation, the text similarity calculation unit is specifically used to:

[0062] Calculating the weight of the similarity of the formula sequence based on the proportion of the first formula sequence and the second formula sequence in the non-symbol sequence;

[0063] The weight of the plain text sequence similarity is calculated based on the proportion of the first plain text sequence and the second plain text sequence in the non-symbol sequence.

[0064] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the text similarity calculation method of the first aspect or any implementation of the first aspect.

[0065] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0066] The memory is used to store computer programs;

[0067] The processor is used to execute the computer program so that the electronic device can implement the text similarity calculation method of the first aspect or any implementation manner of the first aspect.

[0068] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the text similarity calculation method of the above-mentioned first aspect or any implementation of the first aspect.

[0069] By means of the above technical scheme, the present application provides a method and related device for calculating text similarity. After obtaining the first text and the second text to be calculated for similarity; firstly extract diversified sequences from the first text and the second text respectively to obtain at least one group of sequences; then calculate the similarity of each group of sequences respectively; finally, perform weighted summation on the similarity of each group of sequences to calculate the similarity of the first text and the second text. In this scheme, by calculating the similarity of the diversified information in the text respectively, and then fusing the similarities of each diversified information to obtain the final similarity, the accuracy of the similarity calculation of the diversified text can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0071] Figure 1 A flowchart of a text similarity calculation method provided in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of the structure of a text similarity calculation device provided in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] With the rapid development of informatization, text duplication detection technology has been widely used in education, publishing, scientific research and other fields. With the development of technology in various fields, texts in various fields have become more and more diversified. Texts containing multiple languages ​​​​(such as texts containing characters in multiple languages ​​​​) and texts containing different character types (such as symbols, formulas, etc.) (such as science test questions containing complex symbols and formulas) have emerged. The demand for duplicate detection of diversified texts is becoming more urgent.

[0078] At present, text similarity calculation methods are mostly used to check for duplicate texts. However, the existing text similarity calculation methods have the following defects:

[0079] Different word segmentation methods are required for different languages. For example, from a semantic perspective, Chinese uses "characters" as the smallest unit, while English and other languages ​​use "words" as the smallest unit, separated by spaces. Therefore, the existing text similarity calculation method needs to be adapted to the language and has poor universality. For example, the existing text similarity calculation method uses the Jieba word segmentation + LCSS (Longest Common Subsequence) algorithm for Chinese texts; and the LCSS algorithm and its related optimization algorithms for English, French, Japanese and other languages.

[0080] For texts containing formulas, the existing text similarity calculation method often treats the formula as plain text (latex format, etc.) for calculation, and is unable to perform calculations based on the actual features of the formula. For some texts containing complex formulas (such as science test questions), this will interfere with the accuracy of the calculation results.

[0081] In addition, in many text paragraphs, symbols play a key role. For example, in Chinese, after the same text, "?" and "!" may represent a rhetorical question and an exclamation, respectively, with completely different tones. Existing methods for calculating text similarity often treat symbols as text for calculation. For some texts with weak symbolic meanings and many short sentences, the weight of symbols will be too large, resulting in deviations in the calculation results.

[0082] It can be seen that the existing text similarity calculation methods are unable to adapt to the similarity calculation of texts containing multiple languages ​​(such as texts containing characters in multiple languages) and texts containing different character types (such as symbols, formulas, etc.) (such as science test questions containing complex symbols and formulas), resulting in poor accuracy in similarity calculation of diversified texts, and in turn causing deviations in the duplicate checking results of diversified texts.

[0083] Therefore, how to provide a text similarity calculation method to improve the accuracy of similarity calculation of diversified texts has become a technical problem that needs to be solved urgently by those skilled in the art.

[0084] In order to solve the above problems, the present application embodiment provides a text similarity calculation method. The text similarity calculation method of the present application embodiment is described in detail below with reference to the accompanying drawings.

[0085] Reference Figure 1 , Figure 1 A flowchart of a text similarity calculation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a text similarity calculation method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0086] S101: Acquire a first text and a second text to be subjected to similarity calculation;

[0087] In the present application, the first text and the second text to be similarity calculated can be texts of any form in any field, such as papers, science test questions, etc., and the present application does not impose any limitation on this.

[0088] In one achievable manner, the obtaining of the first text and the second text to be subjected to similarity calculation includes: obtaining the first original text and the second original text to be subjected to similarity calculation; and using the first original text and the second original text as the first text and the second text to be subjected to similarity calculation.

[0089] Considering that the original text often contains some abnormal information, this abnormal information will affect the text similarity calculation and even lead to the accuracy of the text similarity calculation results.

[0090] Therefore, in another achievable manner, the obtaining of the first text and the second text for which similarity calculation is to be performed includes: obtaining the first original text and the second original text for which similarity calculation is to be performed; preprocessing the first original text and the second original text to obtain a preprocessed first text and a preprocessed second text, wherein the preprocessed first text and the preprocessed second text are the first text and the second text for which similarity calculation is to be performed.

[0091] As an achievable method, preprocessing the first original text and the second original text may specifically include filtering abnormal characters and redundant spaces of the first original text and the second original text. Of course, other preprocessing methods may also be included, which may be specifically set based on scenario requirements and are not limited in this application.

[0092] S102: performing diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences;

[0093] In the present application, based on the preset diversified information type, diversified sequences can be extracted from the first text and the second text respectively to obtain at least one group of sequences, each group of sequences corresponding to a diversified information type; specifically, the type of diversified information contained in the first text and the type of diversified information contained in the second text can be the same or different. If the first text contains a certain type of diversified information, and the second text does not contain the type of diversified information, then in a group of sequences corresponding to the type of diversified information, the sequence corresponding to the second text is empty. If neither the first text nor the second text contains a certain type of diversified information, then a group of sequences corresponding to the type of diversified information is empty, and the group of sequences will no longer be considered in the future.

[0094] In one achievable manner, the preset diversified information type may include any one or more of formulas, symbols, graphs, and tables in addition to plain text.

[0095] In a possible implementation, a regular matching method may be used to perform diversified sequence extraction on the first text and the second text respectively.

[0096] S103: calculating the similarity of each group of sequences respectively;

[0097] In the present application, the similarity between the plain text sequence and the symbol sequence can be calculated based on LCSS (Longest Common Subsequence), and the similarity between the formula sequence can be calculated based on the edit distance. The details can be described in detail in the following embodiments, and will not be described here.

[0098] S104: Based on the similarities of the sequences, the similarity between the first text and the second text is calculated.

[0099] After obtaining the similarity of each group of sequences, a weighted sum may be performed on the similarity of each group of sequences to calculate the similarity between the first text and the second text, which will be described in detail in the following embodiments and will not be described in detail here.

[0100] The text similarity calculation method and related devices provided in this embodiment. After obtaining the first text and the second text to be similarity calculated; firstly perform diversified sequence extraction on the first text and the second text respectively to obtain at least one group of sequences; then calculate the similarity of each group of sequences respectively; finally perform weighted summation on the similarity of each group of sequences to calculate the similarity of the first text and the second text. In this scheme, by performing similarity calculation on the diversified information in the text respectively, and then integrating the similarities of each diversified information to obtain the final similarity, the accuracy of similarity calculation of diversified texts can be improved.

[0101] Considering that for texts containing formulas, the existing text similarity calculation method often treats the formula as plain text (latex format, etc.) for calculation, and cannot calculate based on the actual features of the formula. For some texts containing complex formulas (such as science test questions), it will interfere with the accuracy of the calculation results. In addition, in many text paragraphs, symbols play a more critical role. For example, in Chinese, after the same text, "?" and "!" may represent rhetorical questions and exclamations, respectively, with completely different tones; the existing text similarity calculation method often treats symbols as text for calculation. For some texts with weak symbol semantics and more short sentences, the weight of symbols will be too large, resulting in deviations in the calculation results.

[0102] Therefore, as a preferred implementation method, the preset diversified information types include plain text, formulas and symbols. The first text and the second text are subjected to diversified sequence extraction respectively, and the obtained sequences may contain symbol sequences and formula sequences in addition to plain text sequences. There may be two cases for at least one group of sequences, one of which does not contain a formula sequence, and the other contains a formula sequence. Among them, the plain text sequence is a sequence composed of all words in the text in order. For Chinese, Japanese, etc., it can be done in units of "characters", and for English and French, it is in units of "words", which solves the problem of universality well and can be adapted to multiple languages. The symbol sequence is a sequence composed of all symbols in the text (such as punctuation, question numbers, option numbers, etc.) in order, and the formula sequence is a sequence composed of all formulas in the text after word segmentation.

[0103] In a possible implementation, if the first text and the second text do not include formulas, diversified sequence extraction is performed on the first text and the second text respectively to obtain at least one set of sequences, including: performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence; and, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence.

[0104] Specifically, regular matching can be performed according to text regularization and symbol regularization in turn, and the plain text and symbols can be extracted and segmented respectively to obtain a plain text sequence and a symbol sequence.

[0105] Based on this, respectively calculating the similarity of each group of sequences includes: calculating the plain text sequence similarity based on the first plain text sequence and the second plain text sequence; and calculating the symbol sequence similarity based on the first symbol sequence and the second symbol sequence.

[0106] It should be noted that the similarity between the plain text sequence and the symbol sequence can be calculated based on LCSS (Longest Common Subsequence).

[0107] For ease of understanding, the following formula can be used to calculate the similarity between the plain text sequence and the symbol sequence:

[0108]

[0109]

[0110] Where text1 is the first plain text sequence, text2 is the second plain text sequence, symbol1 is the first symbol sequence, symbol2 is the second symbol sequence, S text is the similarity of the plain text sequence, S symbol is the similarity of the symbol sequence.

[0111] Based on this, the similarity between the first text and the second text is calculated based on the similarity of each group of sequences, including:

[0112] Determine the weight of the plain text sequence similarity and the weight of the symbol sequence similarity; based on the weight of the plain text sequence similarity and the weight of the symbol sequence similarity, perform weighted summation on the plain text sequence similarity and the symbol sequence similarity to obtain the similarity between the first text and the second text.

[0113] In the present application, the weight of the plain text sequence similarity and the weight of the symbol sequence similarity can be preset, and the distribution of the two can be specified by the specific business scenario. In some scenarios, the symbols have greater distinctiveness, so their weights can be appropriately increased, while some scenarios are not sensitive to symbol differences, so their weights can be appropriately reduced for adaptation.

[0114] For ease of understanding, the similarity between the first text and the second text may be calculated using the following formula:

[0115]

[0116] Among them, S total is the similarity between the first text and the second text, S text is the similarity of the plain text sequence, S symbol is the similarity of the symbol sequence, ω text is the weight of the plain text sequence similarity, ω symbol is the weight of the symbol sequence similarity.

[0117] In another possible implementation, if the first text and the second text include formulas, diversified sequence extraction is performed on the first text and the second text respectively to obtain at least one group of sequences, including: performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence; performing formula sequence extraction on the first text to obtain a first formula sequence; and, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence; and performing formula sequence extraction on the second text to obtain a second formula sequence.

[0118] In one implementation, plain text, formulas, and symbols may be extracted and segmented according to text regularization, symbol regularization, and formula regularization to obtain plain text sequences, symbol sequences, and formula sequences.

[0119] Based on this, the respectively calculating the similarity of each group of sequences includes: calculating the plain text sequence similarity based on the first plain text sequence and the second plain text sequence; calculating the symbol sequence similarity based on the first symbol sequence and the second symbol sequence; calculating the formula sequence similarity based on the first formula sequence and the second formula sequence.

[0120] It should be noted that the similarity between the plain text sequence and the symbol sequence can be calculated based on LCSS (Longest Common Subsequence), and the similarity between the formula sequence can be calculated based on the edit distance.

[0121] For ease of understanding, the following formula can be used to calculate the similarity between the plain text sequence and the symbol sequence:

[0122]

[0123]

[0124] Where text1 is the first plain text sequence, text2 is the second plain text sequence, symbol1 is the first symbol sequence, symbol2 is the second symbol sequence, S text is the similarity of the plain text sequence, S symbol is the similarity of the symbol sequence.

[0125] For ease of understanding, the similarity of the formula sequence can be calculated using the following formula:

[0126]

[0127] Among them, formula1 is the first formula sequence, formula2 is the second formula sequence, S formula is the similarity of the formula sequence.

[0128] Based on this, the similarity between the first text and the second text is calculated based on the similarity of each group of sequences, including:

[0129] S201: Obtaining a preset weight of symbol sequence similarity and a preset weight of non-symbol sequence similarity;

[0130] In the present application, the weight of symbol sequence similarity and the weight of non-symbol sequence similarity are preset, and their distribution can be specified by specific business scenarios. In some scenarios, the symbols have greater distinctiveness, so their weights can be appropriately increased, while some scenarios are insensitive to symbol differences, so their weights can be appropriately reduced for adaptation.

[0131] S202: Calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity;

[0132] As an implementable manner, the calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity include: calculating the weight of the formula sequence similarity based on the proportion of the first formula sequence and the second formula sequence in the non-symbol sequence; calculating the weight of the plain text sequence similarity based on the proportion of the first plain text sequence and the second plain text sequence in the non-symbol sequence.

[0133] For ease of understanding, the weight of the formula sequence similarity and the weight of the plain text sequence similarity can be calculated using the following formula:

[0134]

[0135]

[0136] Among them, ω formula is the weight of formula sequence similarity, ω text is the weight of the plain text sequence similarity, formula1 is the first formula sequence, formula2 is the second formula sequence, text1 is the first plain text sequence, and text2 is the second plain text sequence.

[0137] S203: Based on the weight of the symbol sequence similarity, the weight of the non-symbol sequence similarity, the weight of the formula sequence similarity, and the weight of the plain text sequence similarity, a weighted sum is performed on the formula sequence similarity, the plain text sequence similarity, and the symbol sequence similarity to obtain the similarity between the first text and the second text.

[0138] For ease of understanding, the similarity between the first text and the second text may be calculated using the following formula:

[0139]

[0140] Among them, S total is the similarity between the first text and the second text, S formula is the similarity of the formula sequence, S text is the similarity of the plain text sequence, S symbol is the similarity of the symbol sequence, ω text+formula is the weight of the non-symbolic sequence similarity, ω formula is the weight of formula sequence similarity, ω text is the weight of the plain text sequence similarity, ω symbol is the weight of the symbol sequence similarity.

[0141] A text similarity calculation method provided in an embodiment of the present application is introduced above, and a device for executing the above text similarity calculation method will be introduced below.

[0142] See also Figure 2 , Figure 2 A schematic diagram of the structure of a text similarity calculation device provided in an embodiment of the present application. Figure 2 As shown, the text similarity calculation device includes:

[0143] An acquisition unit 11 is used to acquire a first text and a second text to be subjected to similarity calculation;

[0144] A sequence extraction unit 12, configured to perform diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences;

[0145] A sequence similarity calculation unit 13, used to calculate the similarity of each group of sequences respectively;

[0146] The text similarity calculation unit 14 is used to calculate the similarity between the first text and the second text based on the similarity of each group of sequences.

[0147] In a possible implementation, the acquiring unit is specifically configured to:

[0148] Obtaining a first original text and a second original text to be subjected to similarity calculation;

[0149] The first original text and the second original text are preprocessed to obtain a preprocessed first text and a preprocessed second text, wherein the preprocessed first text and the preprocessed second text are the first text and the second text to be subjected to similarity calculation.

[0150] In a possible implementation, the sequence extraction unit is specifically used to:

[0151] Performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence;

[0152] And, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence.

[0153] In a possible implementation, the sequence similarity calculation unit is specifically used to:

[0154] Calculating a plain text sequence similarity based on the first plain text sequence and the second plain text sequence;

[0155] Symbol sequence similarity is calculated based on the first symbol sequence and the second symbol sequence.

[0156] In a possible implementation, the sequence extraction unit is further specifically configured to:

[0157] Extracting a formula sequence from the first text to obtain a first formula sequence;

[0158] And, extracting a formula sequence from the second text to obtain a second formula sequence.

[0159] In a possible implementation, the sequence similarity calculation unit is further specifically used for:

[0160] The formula sequence similarity is calculated based on the first formula sequence and the second formula sequence.

[0161] In a possible implementation, the text similarity calculation unit is specifically used to:

[0162] Determining a weight for plain text sequence similarity and a weight for symbol sequence similarity;

[0163] Based on the weight of the plain text sequence similarity and the weight of the symbol sequence similarity, a weighted sum is performed on the plain text sequence similarity and the symbol sequence similarity to obtain the similarity between the first text and the second text.

[0164] In a possible implementation, the text similarity calculation unit is specifically used to:

[0165] Obtaining a preset weight of symbol sequence similarity and a preset weight of non-symbol sequence similarity;

[0166] Calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity;

[0167] Based on the weight of the symbol sequence similarity, the weight of the non-symbol sequence similarity, the weight of the formula sequence similarity, and the weight of the plain text sequence similarity, the formula sequence similarity, the plain text sequence similarity, and the symbol sequence similarity are weighted and summed to obtain the similarity between the first text and the second text.

[0168] In a possible implementation, the text similarity calculation unit is specifically used to:

[0169] Calculating the weight of the similarity of the formula sequence based on the proportion of the first formula sequence and the second formula sequence in the non-symbol sequence;

[0170] The weight of the plain text sequence similarity is calculated based on the proportion of the first plain text sequence and the second plain text sequence in the non-symbol sequence.

[0171] The present application also provides an electronic device in an embodiment. Figure 3 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] like Figure 3As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0173] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0174] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any text similarity calculation method provided in the embodiment of the present application.

[0175] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any text similarity calculation method provided in the embodiment of the present application.

[0176] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0177] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0179] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A text similarity calculation method, characterized in that: include: Obtain a first text and a second text to be subjected to similarity calculation; Performing diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences; Calculating the similarity of the sequences in each group respectively; Based on the similarities of the sequences in each group, the similarity between the first text and the second text is calculated.

2. The method according to claim 1, characterized in that The obtaining of the first text and the second text to be subjected to similarity calculation comprises: Obtaining a first original text and a second original text to be subjected to similarity calculation; The first original text and the second original text are preprocessed to obtain a preprocessed first text and a preprocessed second text, wherein the preprocessed first text and the preprocessed second text are the first text and the second text to be subjected to similarity calculation.

3. The method according to claim 1 or 2, characterized in that: The step of performing multivariate sequence extraction on the first text and the second text respectively to obtain at least one set of sequences includes: Performing plain text sequence extraction on the first text to obtain a first plain text sequence; performing symbol sequence extraction on the first text to obtain a first symbol sequence; And, performing plain text sequence extraction on the second text to obtain a second plain text sequence; performing symbol sequence extraction on the second text to obtain a second symbol sequence.

4. The method according to claim 3, characterized in that The respectively calculating the similarity of each group of sequences comprises: Calculating a plain text sequence similarity based on the first plain text sequence and the second plain text sequence; Symbol sequence similarity is calculated based on the first symbol sequence and the second symbol sequence.

5. The method according to claim 3, characterized in that: The extracting multiple sequences from the first text and the second text respectively to obtain at least one set of sequences also includes: Extracting a formula sequence from the first text to obtain a first formula sequence; And, extracting a formula sequence from the second text to obtain a second formula sequence.

6. The method according to claim 5, characterized in that The separately calculating the similarity of each group of sequences further comprises: The formula sequence similarity is calculated based on the first formula sequence and the second formula sequence.

7. The method according to claim 4, characterized in that The calculating the similarity between the first text and the second text based on the similarity of each group of sequences includes: Determining a weight for plain text sequence similarity and a weight for symbol sequence similarity; Based on the weight of the plain text sequence similarity and the weight of the symbol sequence similarity, a weighted sum is performed on the plain text sequence similarity and the symbol sequence similarity to obtain the similarity between the first text and the second text.

8. The method according to claim 5, characterized in that The calculating the similarity between the first text and the second text based on the similarity of each group of sequences includes: Obtaining a preset weight of symbol sequence similarity and a preset weight of non-symbol sequence similarity; Calculating the weight of the formula sequence similarity and the weight of the plain text sequence similarity; Based on the weight of the symbol sequence similarity, the weight of the non-symbol sequence similarity, the weight of the formula sequence similarity, and the weight of the plain text sequence similarity, the formula sequence similarity, the plain text sequence similarity, and the symbol sequence similarity are weighted and summed to obtain the similarity between the first text and the second text.

9. The method according to claim 6, characterized in that The calculating of the weight of the formula sequence similarity and the weight of the plain text sequence similarity includes: Calculating the weight of the similarity of the formula sequence based on the proportion of the first formula sequence and the second formula sequence in the non-symbol sequence; The weight of the plain text sequence similarity is calculated based on the proportion of the first plain text sequence and the second plain text sequence in the non-symbol sequence.

10. A text similarity calculation device, characterized in that: include: An acquisition unit, used for acquiring a first text and a second text to be subjected to similarity calculation; A sequence extraction unit, used for performing diversified sequence extraction on the first text and the second text respectively to obtain at least one set of sequences; A sequence similarity calculation unit, used to calculate the similarity of each group of sequences respectively; The text similarity calculation unit is used to calculate the similarity between the first text and the second text based on the similarity of each group of the sequences.

11. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the text similarity calculation method as claimed in any one of claims 1 to 9.

12. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the text similarity calculation method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the text similarity calculation method as described in any one of claims 1 to 9.

Images