A Multilingual Text Similarity Comparison Method

Abstract

The present application belongs to the field of natural language processing technology, and specifically relates to a method for similarity comparison of multilingual texts, which aims to solve the problem of high difficulty and low efficiency in complex text comparison of multilingual intersections. The method includes: obtaining an undirected graph constructed by comparison dictionaries between different languages and a mapping dictionary constructed by nodes and edges; splitting the first text and the second text into text comparison units; determining the target mapping set corresponding to each term after the text comparison unit is segmented through the undirected graph and the mapping dictionary; determining the candidate comparison unit of any text comparison unit in the first text in the second text through the target mapping set; and performing similarity comparison based on the coverage ratio of hit identifiers in any text comparison unit and the candidate comparison unit. The present application can efficiently realize complex text comparison of multilingual intersections, break down language barriers, avoid malicious cross-language plagiarism, and maintain a globalized academic ecology.

Description

Technical Field

[0001] The present application belongs to the field of natural language processing technology, and specifically relates to a multilingual text similarity comparison method. Background Art

[0002] By comparing the similarity of texts in different languages, we can achieve tasks such as cross-language plagiarism detection and cross-language retrieval. Existing methods for comparing text similarity in different languages mainly utilize the semantic associations between different languages.

[0003] For example, through a deep learning-based machine translation model, the corpus data of two texts in different languages to be compared are aligned to achieve translation from the source language text to the target language text, thereby converting the two texts in different languages into texts in the same language, and then performing a similarity comparison of the texts in the same language.

[0004] For example, based on the aligned corpora of two texts in different languages to be compared, the source text and the target text are segmented and preprocessed respectively, each word or phrase in the source language text is mapped to the corresponding word or phrase in the target language, and then the similarity between the two texts is calculated.

[0005] Aligned corpus-based comparison methods can only compare two languages simultaneously and are not suitable for comparing complex documents with multiple languages. Large-scale multilingual similarity comparisons require extremely high efficiency in text comparisons. Existing deep learning-based methods suffer from long response times, slow speed, and high comparison costs. Summary of the Invention

[0006] In order to solve the above-mentioned problems in the prior art, namely, the high difficulty and low efficiency of complex multilingual text comparison, the first aspect of this application proposes a multilingual text similarity comparison method, which includes:

[0007] Step S10: obtaining an undirected graph constructed based on the multilingual dictionary set, wherein any language in the multilingual dictionary set corresponds to at least one dictionary, the nodes of the undirected graph are words in any language, and the edges of the undirected graph are the comparison relationships between nodes in different languages;

[0008] Step S20, obtaining a mapping dictionary constructed by nodes and a mapping set of nodes, wherein the mapping set includes identifiers of each edge connected to the node;

[0009] Step S30, performing text segmentation on the first text and the second text to be compared, respectively, to generate a plurality of text comparison units;

[0010] Step S40 , segmenting each text comparison unit based on the node to obtain multiple terms, and determining a target mapping set corresponding to any term based on the undirected graph and the mapping dictionary;

[0011] Step S50, based on the target mapping set, determining a candidate comparison unit of any text comparison unit in the first text in the second text, wherein an identifier that is hit in both any text comparison unit and the candidate comparison unit is a hit identifier;

[0012] Step S60, determining a first hit word with a hit identifier in any text comparison unit, and a second hit word with a hit identifier in a candidate comparison unit;

[0013] Step S70 , performing a similarity comparison between any text comparison unit and the candidate comparison unit based on a first coverage ratio of the first hit word in any text comparison unit and a second coverage ratio of the second hit word in the candidate comparison unit.

[0014] In some embodiments, obtaining an undirected graph constructed based on a multilingual dictionary set includes:

[0015] Take words in any language as nodes and connect two nodes that have a mapping relationship based on the mapping dictionary to generate a reference edge;

[0016] Get the number of languages of other nodes connected to any node, as well as the number of base edges connected to any node;

[0017] When the number of languages and the number of reference edges are the same, determine that the other nodes are translation pairs;

[0018] Connect the nodes that are translation pairs to each other to generate extended edges;

[0019] Generate an undirected graph from nodes, base edges, and extension edges.

[0020] In some embodiments, obtaining an undirected graph constructed based on a multilingual dictionary set further includes:

[0021] Determine the translation weight of the base edge;

[0022] The translation weight of the extended edge is determined based on the translation weight of the base edge and a preset attenuation coefficient.

[0023] In some embodiments, when the number of languages and the number of reference edges are the same, determining that other nodes are translation pairs includes:

[0024] When the number of languages and the number of reference edges are the same, and the number of reference edges crossing between other nodes is less than or equal to a preset cross-edge threshold, the other nodes are determined to be translation pairs.

[0025] In some embodiments, determining a candidate alignment unit of any text alignment unit in the first text in the second text based on the target mapping set includes:

[0026] Taking the text comparison unit as a hierarchy, an inverted index is established for the second text according to the identifiers in each target mapping set, and a text comparison unit corresponding to any identifier in the second text is determined;

[0027] Numbering the text comparison units corresponding to each identifier in the second text and forming a unit number set;

[0028] Counting the unit number corresponding to any reference identifier, wherein the reference identifier is any identifier in the target mapping set corresponding to the term of the first text;

[0029] Determine the similarity ratio of any unit number based on the number of times the unit number appears in the statistical process and the corresponding translation weight;

[0030] According to the similarity ratio, candidate alignment units are selected.

[0031] In some embodiments, determining the similarity ratio of any unit number based on the number of times the unit number appears in the statistical process and the corresponding translation weight includes:

[0032] In the case where a unit number appears, the corresponding translation weights are accumulated to obtain the cumulative weight;

[0033] The similarity ratio is calculated as the ratio of the cumulative weight to the number of elements in the unit number set.

[0034] In some embodiments, selecting candidate comparison units based on similarity ratios includes:

[0035] When the similarity ratio of any unit number is greater than or equal to a preset similarity threshold, the text comparison unit corresponding to any unit number is selected as a candidate comparison unit.

[0036] In some embodiments, determining a target mapping set corresponding to any term based on the undirected graph and the mapping dictionary includes:

[0037] In the undirected graph, a node where any term is located and edges connecting the nodes based on the languages of the first text and the second text are selected to form a subgraph;

[0038] Generate a target mapping set based on the nodes in the subgraph and the identifiers of the edges connecting the nodes.

[0039] In some embodiments, obtaining the first coverage ratio and the second coverage ratio includes:

[0040] Obtaining a first union of the first hit word in any text comparison unit, and counting the number of characters in the first union as the first hit coverage length of the first hit word in any text comparison unit;

[0041] Calculate the ratio of the first hit coverage length to the number of characters in any text comparison unit as the first coverage ratio;

[0042] Obtaining a second union of the second hit word in the candidate comparison unit, and counting the number of characters in the second union as a second hit coverage length of the second hit word in the candidate comparison unit;

[0043] The ratio of the second hit coverage length to the number of characters in the candidate alignment unit is calculated as the second coverage ratio.

[0044] In some embodiments, performing a similarity comparison on any text comparison unit and a candidate comparison unit includes:

[0045] When the first coverage ratio of the first hit word in any text comparison unit is greater than the first preset ratio, and the second coverage ratio of the second hit word in the candidate comparison unit is greater than the second preset ratio, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0046] Beneficial effects of this application:

[0047] (1) First, an undirected graph constructed by obtaining the comparison dictionaries between different languages and a mapping dictionary constructed by nodes and edges are used as a tool for similarity comparison between different languages; then, the first text and the second text are split into small text comparison units to facilitate similarity comparison; further, the target mapping set corresponding to each term after the text comparison unit is segmented is determined through the undirected graph and the mapping dictionary. The identifiers of each edge in the target mapping set represent the comparison relationship between the terms in different languages. The identifiers can be used to locate the terms in the first text and the second text respectively; then, the target mapping set is used to determine the candidate comparison unit of any text comparison unit in the first text in the second text, so as to achieve preliminary positioning of similar text segments; finally, similarity comparison is performed based on the coverage ratio of the hit identifiers in any text comparison unit and the candidate comparison unit, thus completing the cross-language text similarity comparison. Since the undirected graph includes multiple languages, this application can realize direct similarity comparison of multilingual texts based on the undirected graph, and achieve complex text comparison across multiple languages without translation conversion, breaking the language barrier in academic creation, avoiding malicious cross-language plagiarism, and maintaining a global academic ecology.

[0048] (2) Since the undirected graph and mapping dictionary are generated and stored in advance, they can be directly called when performing similarity comparison. They do not involve high-parameter models based on deep learning and are basic numerical operations. This improves the efficiency of cross-language text similarity comparison and reduces development costs.

[0049] (3) In addition, undirected graphs can be expanded and assembled by adding or subtracting reference dictionaries without requiring major changes to the method structure. This approach can meet the needs of different users and has strong applicability and a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0051] Figure 1 This is a flowchart of a multilingual text similarity comparison method provided by an embodiment of the present application;

[0052] Figure 2 This is a schematic diagram of an undirected graph provided by an embodiment of the present application;

[0053] Figure 3 This is a flowchart of a multilingual text similarity comparison method provided by an embodiment of the present application;

[0054] Figure 4 This is another undirected graph schematic diagram provided by an embodiment of the present application;

[0055] Figure 5 This is another step flow chart of a multilingual text similarity comparison method provided by an embodiment of the present application;

[0056] Figure 6 This is another step flow chart of a multilingual text similarity comparison method provided by an embodiment of the present application;

[0057] Figure 7 This is another step flow chart of a multilingual text similarity comparison method provided by an embodiment of the present application;

[0058] Figure 8 This is a system block diagram of a multilingual text similarity comparison system provided by an embodiment of the present application;

[0059] Figure 9 It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0061] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] A multilingual text similarity comparison method according to the first embodiment of the present application includes steps S10 to S70. Figure 1 As shown, the method includes:

[0063] Step S10: obtaining an undirected graph constructed based on a multilingual dictionary set, wherein any language in the multilingual dictionary set corresponds to at least one dictionary, the nodes of the undirected graph are words in any language, and the edges of the undirected graph are the comparison relationships between nodes in different languages.

[0064] Alternatively, a bilingual dictionary is a reference tool that arranges words and phrases in two or more languages in a corresponding manner, used for translation and understanding between different languages, such as a Chinese-English dictionary, a Chinese-French dictionary, etc.

[0065] In the embodiment of the present application, the reference dictionary set includes a plurality of reference dictionaries, wherein each language corresponds to at least one reference dictionary.

[0066] As an example, the comparison dictionary set M includes a Chinese-English dictionary A_B, a Chinese-Russian dictionary A_C, an English-Russian dictionary B_C, and a Russian-Italian dictionary C_D, which can be expressed as:

[0067]

[0068] In the above example, Chinese corresponds to the Chinese-English dictionary A_B and the Chinese-Russian dictionary A_C, English corresponds to the Chinese-English dictionary A_B and the English-Russian dictionary B_C, Russian corresponds to the English-Russian dictionary B_C and the Russian-Italian dictionary C_D, and Italian corresponds to the Russian-Italian dictionary C_D.

[0069] Furthermore, a translation comparison undirected graph is constructed based on the comparison dictionary set.

[0070] As an example, the Chinese-English dictionary A_B includes pairs: a1-b1, a1-b2, where a1 is a Chinese word, and b1 and b2 are English words respectively.

[0071] The Chinese-Russian dictionary A_C includes a pair: a2-c3, where a2 is a Chinese word and c3 is a Russian word.

[0072] The English-Russian dictionary B_C includes the following pairs: b1-c1, b2-c2, b2-c3, where c1, c2 and c3 are each a Russian word.

[0073] The Russian-Italian dictionary C_D includes the following pairs: c1-d3, c2-d1, c3-d2, c3-d5, where d1, d2, d3 and d5 are each an Italian word.

[0074] It is understandable that the Chinese word a1 can form a pair with the English words b1 and b2 respectively, that is, the Chinese word a1 can be translated into the English words b1 and b2 respectively.

[0075] Furthermore, an undirected graph of the reference dictionary set is generated by taking words in any language as nodes and reference relationships between nodes in different languages as edges.

[0076] As an example, based on the comparison pairs included in each comparison dictionary in the comparison dictionary set M, the following is generated: Figure 2 In the undirected graph shown, each edge is labeled with a number to facilitate edge identification.

[0077] In other embodiments, the comparison dictionary may also include different variants of the same language, such as the comparison between simplified Chinese and traditional Chinese, or the comparison of professional terms in different fields.

[0078] Step S20: Obtain a mapping dictionary constructed by nodes and their mapping sets, wherein the mapping set includes identifiers of edges connected to the nodes.

[0079] Optionally, for any node, there is at least one edge connected to it, such as Figure 2 As shown, taking node a1 as an example, node b1 and node b2 are connected in the undirected graph, the edge between node a1 and node b1 is identified as 0, and the edge between node a1 and node b2 is identified as 2, then the mapping set of node a1 is [0,2].

[0080] Furthermore, the mapping set of each node is obtained and a mapping dictionary is constructed with the corresponding node. Figure 2 As an example of an undirected graph, the resulting mapping dictionary is:

[0081] {

[0082] a1:[0,2]

[0083] a2:[3]

[0084] b1:[0,1]

[0085] b2:[2,4,5]

[0086] c1:[1,8]

[0087] c2:[4,6]

[0088] c3:[3,5,7,9]

[0089] d1:[6]

[0090] d2:[7]

[0091] d3:[8]

[0092] d5:[9]

[0093] }

[0094] It should be noted that the mapping set of a node is a statistical identification of the edges connected to the node, and the mapping dictionary composed of each node and its mapping set is an icon conversion of the undirected graph, which is convenient for searching and positioning.

[0095] Step S30 : performing text segmentation on the first text and the second text to be compared respectively to generate a plurality of text comparison units.

[0096] Alternatively, text segmentation is the process of dividing continuous text into smaller, more meaningful paragraphs or units according to certain rules or standards. It plays an important role in natural language processing, document processing, information retrieval and other fields.

[0097] In an embodiment of the present application, text segmentation is performed according to actual needs, for example, text segmentation is performed by sentence, fragment, paragraph, chapter, etc., and different text segmentation tools are selected according to different rules.

[0098] It should be noted that the first text and the second text can be in different languages or in the same language, that is, this application can perform text similarity comparison across languages as well as text similarity comparison in the same language.

[0099] As an example, the first text L in Chinese is segmented according to paragraphs to obtain a set of text comparison units {L0, L1, ..., L N}, where L0, L1, ..., L N They are different text comparison units; the second text R in English is segmented according to paragraphs to obtain a text comparison unit set {R0, R1, ..., R N}, where R0, R1, ..., R N They are different text comparison units.

[0100] Step S40 , segmenting each text comparison unit based on the node to obtain multiple terms, and determining a target mapping set corresponding to any term based on the undirected graph and the mapping dictionary.

[0101] Optionally, for any text comparison unit of the first text and the second text, word segmentation is achieved by matching any text comparison unit with a node.

[0102] For example, consider the original text of a text comparison unit: He came to Shanghai Jiao Tong University. The corresponding words for the node include: He, came, Shanghai, Jiao Tong, University, and Shanghai Jiao Tong University. By matching the node, the segmentation result is: He / came / Shanghai / Shanghai Jiao Tong University / Jiao Tong / University. The resulting terms after segmentation include: He, came, Shanghai, Jiao Tong, University, and Shanghai Jiao Tong University.

[0103] It should be noted that the above example is only a simple example. In other embodiments, the text comparison unit for word segmentation may be a longer or shorter paragraph.

[0104] Furthermore, by determining the identifiers of the edges connecting the nodes corresponding to any term in the undirected graph, it is possible to determine the target mapping set corresponding to any term.

[0105] As an example, by searching the undirected graph, it is determined that the edge connected to the node corresponding to "he" in the undirected graph is identified as 0, the edge connected to the node corresponding to "come" in the undirected graph is identified as 1, the edge connected to the node corresponding to "Shanghai" in the undirected graph is identified as 2, the edge connected to the node corresponding to "Shanghai Jiaotong University" in the undirected graph is identified as 3, the edge connected to the node corresponding to "transportation" in the undirected graph is identified as 4, and the edge connected to the node corresponding to "university" in the undirected graph is identified as 5.

[0106] Then the target mapping set corresponding to the term "he" is {0}, the target mapping set corresponding to the term "come to" is {1}, the target mapping set corresponding to the term "Shanghai" is {2}, the target mapping set corresponding to the term "Shanghai Jiaotong University" is {3}, the target mapping set corresponding to the term "transportation" is {4}, and the target mapping set corresponding to the term "university" is {5}.

[0107] Step S50 : determining a candidate comparison unit of any text comparison unit in the first text in the second text based on the target mapping set, wherein an identifier that is hit in both any text comparison unit and the candidate comparison unit is a hit identifier.

[0108] Optionally, an undirected graph edge connects two nodes that have a comparison relationship. After determining multiple target mapping sets included in each text comparison unit, for any text comparison unit in the first text, the corresponding candidate comparison unit can be found in the second text through the identifiers of each edge in the target mapping set.

[0109] As an example, for any text comparison unit to be selected in the second text, the identifiers that are hit in any text comparison unit and the comparison unit to be selected are counted, and the proportion of the number of identifiers included in the comparison unit to be selected is counted. When the proportion is greater than a certain threshold, the text comparison unit to be selected is determined to be a candidate comparison unit of any text comparison unit.

[0110] Furthermore, the identifier corresponding to the term that is hit in any text comparison unit and candidate comparison unit is recorded as a hit identifier.

[0111] It can be understood that the term corresponding to the hit identifier is an identifier that exists in both text comparison units and is the basis for similarity comparison between the two text comparison units.

[0112] In an embodiment of the present application, the hit identifier is the identifier of the edge that exists in any text comparison unit and the candidate comparison unit, that is, the number of hit identifiers is less than or equal to the number of identifiers included in the candidate comparison unit, and is also less than or equal to the number of identifiers in any text comparison unit.

[0113] Step S60: determining a first hit word with a hit identifier in any text comparison unit and a second hit word with a hit identifier in a candidate comparison unit.

[0114] Optionally, through the one-to-one correspondence between edges and nodes, the corresponding terms can be determined in any text comparison unit and candidate comparison unit respectively through identification, that is, the first hit word in any text comparison unit and the second hit word in the candidate comparison unit.

[0115] Step S70 , performing a similarity comparison between any text comparison unit and the candidate comparison unit based on a first coverage ratio of the first hit word in any text comparison unit and a second coverage ratio of the second hit word in the candidate comparison unit.

[0116] Optionally, the more coverage a hit word has in the text, the more corresponding similar terms there are. By calculating the coverage length of the hit word in two text comparison units, the hit ratio of the hit word can be determined, which is used to determine the degree of similarity between the two text comparison units.

[0117] As an example, when both the first coverage ratio and the second coverage ratio are large, that is, there are many similar terms in the two text comparison units, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0118] For example, a similarity threshold is set, and when both the first coverage ratio and the second coverage ratio are greater than or equal to the similarity threshold, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0119] In order to more clearly illustrate the multilingual text similarity comparison method of the present application, each step in the embodiment of the present application is described in detail below.

[0120] As a possible implementation, please refer to Figure 3 The process of constructing an undirected graph in step S10 includes the following steps:

[0121] Step S101 : taking words in any language as nodes, connecting two nodes that have a cross-reference relationship based on a cross-reference dictionary to generate a reference edge.

[0122] As an example, the comparison dictionary set M is generated as follows Figure 2 The undirected graph shown.

[0123] Generate as Figure 2 The process of the undirected graph shown can be referred to the relevant description in step S10, which will not be repeated here.

[0124] Furthermore, the translation weight of the reference edge is determined.

[0125] It should be noted that the two nodes connected by the base edge are two words in different languages that can be directly translated. The comparison relationship is the most accurate, and the translation weight of the base edge is determined as a translation reference.

[0126] As an example, the translation weight of the base edge is set to 1.

[0127] Step S102: Obtain the number of languages of other nodes connected to any node, and the number of reference edges connected to any node.

[0128] by Figure 2 Take node b1 in the example. The other nodes connected to node b1 are a1 and c1. Among them, the language of a1 is Chinese, the language of c1 is Russian, and the number of languages of other nodes connected to node b1 is 2. There are two base edges connected to node b1, with corresponding identifiers 0 and 1 respectively, that is, the number of base edges is also 2.

[0129] by Figure 2Take node c3 in the example as another example. The other nodes connected to node c3 are a2, b2, d2, and d5. Among them, the language of a2 is Chinese, the language of b2 is English, and the languages of d2 and d5 are Italian. The number of languages of the other nodes connected to node c3 is 3; there are 4 base edges connected to node c3, and the corresponding identifiers are 3, 5, 7, and 9, that is, the number of base edges is 4.

[0130] Step S103: When the number of languages and the number of reference edges are the same, determine that other nodes are translation pairs.

[0131] It is understandable that, when the number of languages and the number of reference edges are the same, several nodes connected by a node can be translated into the same meaning, and mutual translation of several other nodes can be achieved through the node.

[0132] by Figure 2 Taking node b1 in as an example, the number of languages of other nodes connected to node b1 is 2, and the number of base edges is also 2. It is determined that node a1 and node c1 are a translation pair.

[0133] by Figure 2 Taking node c3 in the figure as another example, the number of languages ​​connected to node c3 is 3, and the number of base edges is 4. The number of languages ​​and the number of base edges are different. Nodes a2, b2, d2 and d5 cannot be translation pairs with each other.

[0134] In other embodiments, partial identification can also be performed, that is, the number of languages of some other nodes connected to the node and the corresponding number of reference edges are compared. If the number of languages and the number of reference edges are the same, then some other nodes can be translation pairs with each other.

[0135] Likewise Figure 2 Taking node c3 in as an example, if some of the nodes connected to node c3 are a2 and b2, the language of a2 is Chinese, the language of b2 is English, the number of languages is 2, and the number of base edges connecting nodes a2 and b2 with node c3 is also 2, then nodes a2 and b2 can be translation pairs for each other.

[0136] Furthermore, the extension of translation pairs can be restricted by presetting a cross-edge threshold.

[0137] Optionally, when the number of languages is the same as the number of reference edges, and the number of reference edges crossing between other nodes is less than or equal to a preset cross-edge threshold, it is determined that the other nodes are translation pairs.

[0138] by Figure 2For example, the possible translation pairs are: a1-b1-c1, b1-c1-d3, b2-c2-d1, and a2-c3-b2. After merging, we can get: a1-b1-c1-d3, b2-c2-d1, and a2-c3-b2. Without limiting the number of cross-edges, nodes a1 and d3 can translate each other. However, if the number of cross-edges is limited, for example, with a preset cross-edge threshold of 2, the number of cross-reference edges between nodes a1 and d3 is 3, and nodes a1 and d3 cannot be a translation pair. Alternatively, with a preset cross-edge threshold of 3, nodes a1 and d3 can be a translation pair and translate each other.

[0139] In other embodiments, the value of the preset edge crossing threshold may be set according to actual operating conditions.

[0140] Step S104: Connect the nodes that are translation pairs to each other to generate extended edges.

[0141] In the embodiment of the present application, the preset cross-edge threshold is 3, and the nodes that are translation pairs are connected to each other to generate an extended edge. For example, through a1-b1-c1, the node a1 and the node c1 are connected to each other to generate an extended edge.

[0142] It is understood that the nodes connected by the extended edge are not two directly translated words, but rather mutual translations achieved through multiple dictionary conversions. The extended edge allows for preliminary translation comparison of reference dictionaries not included in the reference dictionary set, thereby expanding the scope of text similarity comparison. For example, for Chinese-English dictionaries not included in the reference dictionary set M, the method of this application can be used to achieve cross-language similarity comparison.

[0143] Furthermore, based on the translation weight of the reference edge and a preset attenuation coefficient, the translation weight of the extended edge is determined.

[0144] Optionally, for newly added translation pairs connected by extended edges, relay translation is performed through multiple comparison dictionaries, and there may be a loss in translation accuracy. Therefore, an equivalent attenuation coefficient is set to redetermine the translation weight of the extended edge.

[0145] In the embodiment of the present application, the attenuation coefficient is set to , the number of cross-edges between two nodes connected by the extended edge that cross the base edge is , then the translation weight of the extended edge is , where 1 is the translation weight of the base edge.

[0146] As an example, setting the attenuation coefficient is 0.5, the number of cross edges between nodes a1 and c1 is 2, then the translation weight of the extended edge connecting node a1 and node c1 is 1*0.5 / 2=0.25.

[0147] Step S105: Generate an undirected graph using the nodes, reference edges, and extended edges.

[0148] Optionally, by Figure 2 Add extended edges to the undirected graph shown in , and then adjust the position to generate the final undirected graph, as shown in Figure 4 shown.

[0149] As a possible implementation, please refer to Figure 5 In step S40, a target mapping set corresponding to any term is determined based on the undirected graph and the mapping dictionary, including the following steps:

[0150] Step S401 : Select any node where a term is located in an undirected graph, and edges connecting the nodes based on the languages of the first text and the second text, to form a subgraph.

[0151] Optional, such as Figure 4 The undirected graph shown includes nodes in multiple languages. By filtering the nodes where any word item is located after word segmentation and the edges that only include the languages of the first and second texts, a subgraph is extracted to serve as a tool for text similarity comparison.

[0152] As an example, the language of the first text is Chinese and the language of the second text is English. Figure 4 The undirected graph shown is extracted, and the resulting subgraph only retains nodes a1, a2, b1, b2, and edges marked as 0, 2, and 14.

[0153] It should be noted that the embodiment of the present application is a simple example. In actual use, the scope of the undirected graph includes all words in the standard dictionary, and vocabulary such as professional terms in different fields can also be added.

[0154] Step S402: Generate a target mapping set based on the nodes in the subgraph and the identifiers of the edges connecting the nodes.

[0155] Optionally, the target mapping set corresponding to any node in the subgraph is determined by searching the undirected graph.

[0156] For example, for the subgraph in step S501, the target mapping set corresponding to node a1 is {0, 2}, the target mapping set corresponding to node a2 is {14}, the target mapping set corresponding to node b1 is {0}, and the target mapping set corresponding to node b2 is {2, 14}.

[0157] As a possible implementation, please refer to Figure 6The process of determining a candidate comparison unit of any text comparison unit in the first text in the second text in step S50 includes the following steps:

[0158] Step S501 : Taking text comparison units as a hierarchy, an inverted index is established for the second text according to the identifiers in each target mapping set, and a text comparison unit corresponding to any identifier in the second text is determined.

[0159] Optionally, an inverted index, also called a reverse index, improves retrieval efficiency by converting the "document-word" format into the "word-document" format.

[0160] In the embodiment of the present application, the format of "text comparison unit-identification" is transformed into the format of "identification-text comparison unit".

[0161] As an example, the second text R is divided into four text comparison units {R0, R1, R2, R3}, and the identifier set corresponding to each text comparison unit generates a text comparison unit-identifier comparison table, as shown in Table 1:

[0162] Table 1

[0163]

[0164] By creating an inverted index for each identifier in the identifier set, a identifier-text comparison unit comparison table is generated, as shown in Table 2:

[0165] Table 2

[0166]

[0167] The text comparison unit where any logo is located can be quickly located through the logo-text comparison unit comparison table.

[0168] Step S502: number the text comparison units corresponding to each identifier in the second text and form a unit number set.

[0169] As an example, according to the identifier-text comparison unit comparison table shown in step S501, the unit number set corresponding to any identifier can be obtained. For example, the unit number set corresponding to identifier 0 is {R0}, the unit number set corresponding to identifier 1 is {R0}, and the unit number set corresponding to identifier 2 is {R0, R1}.

[0170] Step S503 : counting the unit number corresponding to any reference identifier, wherein the reference identifier is any identifier in the target mapping set corresponding to the term of the first text.

[0171] Optionally, the identifier represents a comparison relationship between two languages, and the identifier can be used to determine the corresponding terms in the first text and the second text.

[0172] As an example, if the reference identifier in the first text is {2,3,6,10}, then through the identifier-text comparison unit comparison table, the unit number corresponding to any reference identifier can be obtained: reference identifier 2: {R0 R1}, reference identifier 3: {R0R1 R2}, reference identifier 6: {R1 R2}, reference identifier 10: {R3}.

[0173] By determining the text comparison unit corresponding to the reference identifier in the second text, the location where the similar text exists can be preliminarily located.

[0174] Step S504 : determining the similarity ratio of any unit number based on the number of times the unit number appears in the statistical process and the corresponding translation weight.

[0175] Optionally, when a unit number appears, the corresponding translation weights are accumulated to obtain a cumulative weight; and the ratio of the cumulative weight to the number of elements in the unit number set is calculated as the similarity ratio.

[0176] The calculation formula for the similarity ratio S of any unit number is as follows:

[0177]

[0178] Among them, H represents the number of times any unit number appears in the statistical process, represents the translation weight corresponding to the kth occurrence of any unit number, Indicates the cumulative weight of any unit number, Indicates the number of elements in the unit number set.

[0179] For example, for reference identifier 2: {R0 R1}, reference identifier 3: {R0 R1 R2}, reference identifier 6: {R1 R2}, reference identifier 10: {R3}, the number of elements in the unit number set {R0 R1R2 R3} is 4, the number of unit number R0 appears 2 times, the number of unit number R1 appears 3 times, the number of unit number R2 appears 2 times, and the number of unit number R3 appears 1 times. Assuming that the translation weight corresponding to each occurrence of any unit number is 1, the similarity ratio of unit number R0 is , similarity ratio of unit number R1 , similarity ratio of unit number R2 , similar ratio of unit number R3 .

[0180] It should be noted that the similarity ratio represents the similarity ratio between the text comparison unit corresponding to any unit number and any text comparison unit in the first text. When the similarity ratio is large, it means that the text comparison unit may have text similarity with any text comparison unit in the first text.

[0181] Step S505: Select candidate comparison units based on the similarity ratio.

[0182] Optionally, when the similarity ratio of any unit number is greater than or equal to a preset similarity threshold, the text comparison unit corresponding to any unit number is selected as a candidate comparison unit.

[0183] As an example, the preset similarity threshold may be set to 0.7. In other embodiments, the value may be set according to actual conditions.

[0184] Among them, the similarity ratio of unit number R1 If 0.75>0.7, the text comparison unit corresponding to the unit number R1 is listed as a candidate comparison unit.

[0185] As a possible implementation, please refer to Figure 7 In step S70, the process of determining the first coverage ratio and the second coverage ratio and performing similarity comparison between any text comparison unit and the candidate comparison unit includes the following steps:

[0186] Step S701: Obtain a first union of the first hit word in any text comparison unit, and count the number of characters in the first union as a first hit coverage length of the first hit word in any text comparison unit.

[0187] Optionally, the first hit word may be repeated. For example, for the sample text "He came to Shanghai Jiao Tong University", the first hit word may include both "Shanghai" and "Shanghai Jiao Tong University". Therefore, first obtain the first union of the first hit word in any text comparison unit as the hit character set, and count the number of characters in the first union as the first hit coverage length of the first hit word in any text comparison unit.

[0188] The first union is obtained by determining the original text position corresponding to the hit identifier.

[0189] As an example, the above text comparison unit "He came to Shanghai Jiao Tong University" has a total of 9 characters, and the character positions in the original text are located by digital numbering, that is, each character is located by numbers 0-8.

[0190] Then a word item-identifier-original text position record table can be generated, as shown in Table 3:

[0191] Table 3

[0192]

[0193] By obtaining the union of the original text positions, the number of hit characters, that is, the first hit coverage length, is determined.

[0194] As shown in Table 3, the hit identifiers are 0, 1, 2, 3, and 5, and a table of hit identifiers mapped to original text positions is generated, as shown in Table 4 below:

[0195] Table 4

[0196]

[0197] As shown in Table 4, the original text position of the first hit word is [0,1][1,3][3,5][3,9][7,9]. The first union is obtained by taking the union of the original text positions [0,9].

[0198] In other embodiments, a weight may be assigned to any hit word, and the translation weight may be used as the weight of the hit word. The lengths of the hit words may be weighted and summed to obtain the first hit coverage length.

[0199] Step S702 , calculating the ratio of the first hit coverage length to the number of characters in any text comparison unit as the first coverage ratio.

[0200] Optionally, the first union represents the hit character length of the hit identifier in any text comparison unit. By calculating the ratio of the first hit coverage length to the number of characters in any text comparison unit, the hit ratio of the hit identifier in any text comparison unit can be determined as the first coverage ratio.

[0201] As an example, as shown in Table 4, the first coverage ratio .

[0202] Step S703: Obtain a second union of the second hit word in the candidate comparison unit, and count the number of characters in the second union as a second hit coverage length of the second hit word in the candidate comparison unit.

[0203] Optionally, the second hit word may be repeated. For example, for the sample text "he Come to shanghaijiao tong University", the second hit word may have both "shanghai" and "shanghai jiao tongUniversity". Therefore, first obtain the second union of the second hit word in the candidate comparison unit as the hit character set, and count the number of characters in the second union as the second hit coverage length of the second hit word in the candidate comparison unit.

[0204] The second union is obtained by determining the original text position corresponding to the hit identifier.

[0205] As an example, the above text comparison unit "he come to shanghai jiao tong University" has a total of 40 characters (including spaces), and the character positions in the original text are located by numerical numbering, that is, each character is located by numbers 0-39.

[0206] Then a word item-identifier-original text position record table can be generated, as shown in Table 5:

[0207] Table 5

[0208]

[0209] By obtaining the union of the original text positions, the number of hit characters, that is, the second hit coverage length, is determined.

[0210] As shown in Table 5, the hit identifiers are 0, 1, 2, 3, and 5, and a table of hit identifiers mapped to original text positions is generated, as shown in Table 6 below:

[0211] Table 6

[0212]

[0213] As shown in Table 6, the original text position of the first hit word is [0,2][3,10][11,19][11,40][30,40]. The second union is obtained by taking the union of the original text positions [0,2][3,10][11,40]. The second union is represented by P in Table 6.

[0214] Step S704 : Calculate the ratio of the second hit coverage length to the number of characters in the candidate alignment unit as the second coverage ratio.

[0215] Optionally, the second union represents the hit character length of the hit identifier in the candidate comparison unit. By calculating the ratio of the second hit coverage length to the number of characters in the candidate comparison unit, the hit ratio of the hit identifier in the candidate comparison unit can be determined as the second coverage ratio.

[0216] As an example, as shown in Table 6, the second coverage ratio .

[0217] Step S705, when the first coverage ratio of the first hit word in any text comparison unit is greater than the first preset ratio, and the second coverage ratio of the second hit word in the candidate comparison unit is greater than the second preset ratio, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0218] Optionally, the more coverage a hit word has in the text, the more corresponding similar terms there are. By calculating the coverage length of the hit word in two text comparison units, the hit ratio of the hit word can be determined, which is used to determine the degree of similarity between the two text comparison units.

[0219] As an example, when both the first coverage ratio and the second coverage ratio are large, that is, there are many similar terms in the two text comparison units, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0220] For example, a first similarity threshold and a second similarity threshold are set, and when the first coverage ratio is greater than or equal to the first similarity threshold and the second coverage ratio is greater than or equal to the second similarity threshold, it is determined that any text comparison unit is similar to the candidate comparison unit.

[0221] In the embodiment of the present application, the first similarity threshold and the second similarity threshold are both set to 0.7, then and , then any text comparison unit is similar to the candidate comparison unit.

[0222] In an embodiment of the present application, an undirected graph constructed by obtaining a comparison dictionary between different languages and a mapping dictionary constructed by nodes and edges are first used as a tool for similarity comparison between different languages; then, the first text and the second text are split into small text comparison units to facilitate similarity comparison; further, a target mapping set corresponding to each term after the text comparison unit is segmented is determined through the undirected graph and the mapping dictionary, and the identifier of each edge in the target mapping set represents the comparison relationship between the terms in different languages. The identifier can be used to locate the terms in the first text and the second text respectively; then, the target mapping set is used to determine the candidate comparison unit of any text comparison unit in the first text in the second text, so as to achieve preliminary positioning of similar text segments; finally, a similarity comparison is performed based on the coverage ratio of the hit identifiers in any text comparison unit and the candidate comparison unit, thereby completing the cross-language text similarity comparison. Since the undirected graph includes multiple languages, this application can realize direct similarity comparison of multilingual texts based on the undirected graph, and achieve complex text comparison across multiple languages without translation conversion, breaking the language barrier in academic creation, avoiding malicious cross-language plagiarism, and maintaining a global academic ecology.

[0223] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of this application.

[0224] The multilingual text similarity comparison system of the second embodiment of the present application is as follows: Figure 8 As shown, it includes: an undirected graph acquisition module 100, a mapping dictionary acquisition module 200, a text segmentation module 300, a word segmentation module 400, a candidate comparison unit determination module 500, a hit word determination module 600 and a similarity comparison module 700.

[0225] An undirected graph acquisition module 100 is configured to acquire an undirected graph constructed based on a multilingual dictionary set, wherein each language in the multilingual dictionary set corresponds to at least one dictionary, the nodes of the undirected graph are words in any language, and the edges of the undirected graph are the mapping relationships between nodes in different languages;

[0226] A mapping dictionary acquisition module 200 is used to acquire a mapping dictionary constructed by nodes and a mapping set of nodes, wherein the mapping set includes identifiers of edges connected to the nodes;

[0227] A text segmentation module 300 is used to segment the first text and the second text to be compared respectively to generate a plurality of text comparison units;

[0228] The word segmentation module 400 is used to segment each text comparison unit based on the node to obtain multiple terms, and determine the target mapping set corresponding to each term based on the undirected graph and the mapping dictionary;

[0229] A candidate comparison unit determination module 500 is configured to determine, based on a target mapping set, a candidate comparison unit of any text comparison unit in the first text in the second text, wherein an identifier that is a hit in both any text comparison unit and the candidate comparison unit is a hit identifier;

[0230] A hit word determination module 600 is used to determine a first hit word whose hit identifier is in any text comparison unit, and a second hit word whose hit identifier is in a candidate comparison unit;

[0231] The similarity comparison module 700 is used to perform similarity comparison between any text comparison unit and candidate comparison units based on a first coverage ratio of the first hit word in any text comparison unit and a second coverage ratio of the second hit word in the candidate comparison unit.

[0232] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0233] It should be noted that the multilingual text similarity comparison system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present application can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules or steps and are not considered to be improper limitations on the present application.

[0234] An electronic device according to a third embodiment of the present application includes:

[0235] at least one processor; and

[0236] a memory communicatively connected to at least one of the processors; wherein,

[0237] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned multilingual text similarity comparison method.

[0238] A fourth embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are configured to be executed by the computer to implement the above-mentioned multilingual text similarity comparison method.

[0239] A computer program product according to a fifth embodiment of the present application, when running on an electronic device, enables the electronic device to execute the above-mentioned multilingual text similarity comparison method.

[0240] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the electronic device and computer-readable storage medium described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0241] Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0242] Reference below Figure 9 , which shows a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 9 The server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0243] like Figure 9 As shown, the computer system includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 into the random access memory (RAM) 903. Various programs and data required for system operation are also stored in the RAM 903. The CPU 901, ROM 902 and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0244] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output section 907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 910 as needed so that a computer program read therefrom can be installed into the storage section 908 as needed.

[0245] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0246] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0247] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0248] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0249] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0250] Thus far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A multilingual text similarity comparison method, characterized in that: The method comprises: Step S10: obtaining an undirected graph constructed based on a multilingual dictionary set, wherein each language in the multilingual dictionary set corresponds to at least one dictionary, the nodes of the undirected graph are words in any language, and the edges of the undirected graph are comparison relationships between nodes in different languages; Step S20: obtaining the node and a mapping dictionary constructed by a mapping set of the node, wherein the mapping set includes identifiers of each edge connected to the node; Step S30, performing text segmentation on the first text and the second text to be compared, respectively, to generate a plurality of text comparison units; Step S40, performing word segmentation on each text comparison unit based on the node to obtain multiple terms, and determining a target mapping set corresponding to any term based on the undirected graph and the mapping dictionary; Step S50, based on the target mapping set, determining a candidate comparison unit of any text comparison unit in the first text in the second text, wherein an identifier that is hit in both the any text comparison unit and the candidate comparison unit is a hit identifier; Step S60, determining a first hit word of the hit identifier in any text comparison unit, and a second hit word of the hit identifier in the candidate comparison unit; Step S70 , performing a similarity comparison between the any text comparison unit and the candidate comparison unit according to a first coverage ratio of the first hit word in the any text comparison unit and a second coverage ratio of the second hit word in the candidate comparison unit.

2. A multilingual text similarity comparison method according to claim 1, characterized in that: The obtaining of an undirected graph constructed based on a multilingual dictionary set includes: Taking words in any language as nodes, connecting two nodes that have a mapping relationship based on the mapping dictionary to generate a reference edge; Obtain the number of languages of other nodes connected to any node, and the number of base edges connected to the node; When the number of the languages is the same as the number of the reference edges, determining that the other nodes are translation pairs; Connect the nodes that are translation pairs to each other to generate extended edges; The undirected graph is generated by the nodes, the base edges, and the extended edges.

3. A multilingual text similarity comparison method according to claim 2, characterized in that: The step of obtaining an undirected graph constructed based on a multilingual dictionary set further includes: determining a translation weight of the reference edge; The translation weight of the extended edge is determined based on the translation weight of the reference edge and a preset attenuation coefficient.

4. A multilingual text similarity comparison method according to claim 2, characterized in that: When the number of the languages is the same as the number of the reference edges, determining that the other nodes are translation pairs includes: When the number of languages is the same as the number of reference edges, and the number of reference edges crossing between the other nodes is less than or equal to a preset crossing-edge threshold, it is determined that the other nodes are translation pairs.

5. The multilingual text similarity comparison method according to claim 3, characterized in that: The determining, based on the target mapping set, a candidate comparison unit of any text comparison unit in the first text in the second text includes: Taking text comparison units as a hierarchy, establishing an inverted index for the second text according to the identifiers in each target mapping set, and determining a text comparison unit corresponding to any identifier in the second text; Numbering the text comparison units corresponding to each identifier in the second text and forming a unit number set; Counting the unit number corresponding to any reference identifier, wherein the reference identifier is any identifier in the target mapping set corresponding to the term of the first text; Determining a similarity ratio of any unit number based on the number of times the unit number appears in the statistical process and the corresponding translation weight; The candidate comparison unit is selected according to the similarity ratio.

6. A multilingual text similarity comparison method according to claim 5, characterized in that: Based on the number of occurrences of the unit number in the statistical process and the corresponding translation weight, the similarity ratio of any unit number is determined, including: In the case where the unit number appears, the corresponding translation weights are accumulated to obtain a cumulative weight; The ratio of the cumulative weight to the number of elements in the unit number set is calculated as the similarity ratio.

7. The multilingual text similarity comparison method according to claim 5, characterized in that: The selecting the candidate comparison unit according to the similarity ratio includes: In the case that the similarity ratio of any unit number is greater than or equal to a preset similarity threshold, the text comparison unit corresponding to any unit number is selected as the candidate comparison unit.

8. The multilingual text similarity comparison method according to claim 1, characterized in that: The determining a target mapping set corresponding to any term based on the undirected graph and the mapping dictionary includes: Selecting nodes where any one of the terms is located in the undirected graph, and edges connecting the nodes based on the languages of the first text and the second text, to form a subgraph; The target mapping set is generated based on the nodes in the subgraph and the identifiers of the edges connecting the nodes.

9. The multilingual text similarity comparison method according to claim 1, characterized in that: Obtaining the first coverage ratio and the second coverage ratio includes: Obtaining a first union of the first hit word in any of the text comparison units, and counting the number of characters in the first union as a first hit coverage length of the first hit word in any of the text comparison units; Calculating a ratio of the first hit coverage length to the number of characters in any text comparison unit as the first coverage ratio; Obtaining a second union of the second hit word in the candidate comparison unit, and counting the number of characters in the second union as a second hit coverage length of the second hit word in the candidate comparison unit; The ratio of the second hit coverage length to the number of characters in the candidate alignment unit is calculated as the second coverage ratio.

10. The multilingual text similarity comparison method according to claim 1, characterized in that: The performing similarity comparison between any one of the text comparison units and the candidate comparison units includes: When the first coverage ratio of the first hit word in any text comparison unit is greater than the first preset ratio, and the second coverage ratio of the second hit word in the candidate comparison unit is greater than the second preset ratio, it is determined that any text comparison unit is similar to the candidate comparison unit.

Images