A text similarity determination method, device and equipment and storage medium

Abstract

The application discloses a text similarity determination method, device and equipment and a storage medium. The method comprises the following steps: obtaining a plurality of texts to be processed, performing word segmentation processing on each text to obtain a plurality of word segmentation results corresponding to each text respectively; determining CoNLL format data corresponding to each word segmentation result, and constructing a dependency grammar tree corresponding to each text according to the CoNLL format data corresponding to each text respectively; generating a dependency matrix matched with each text according to the dependency grammar tree corresponding to each text respectively, and determining a target similarity between the plurality of texts according to the dependency matrices. The technical scheme of the embodiment of the application can reduce the complexity of the text similarity determination method and improve the accuracy of the text similarity determination result.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for determining text similarity. Background Technology

[0002] With the rapid expansion of internet data, electronic text has also experienced explosive growth. Among this massive amount of text, quickly retrieving relevant information through fuzzy text search has become a key development trend for selecting important information.

[0003] In existing technologies, when determining text matching information, the similarity between texts is usually determined based on the edit distance between them; or based on neural network models and machine learning methods.

[0004] However, text similarity determination methods based on edit distance only perform simple string matching in the text and cannot represent the implicit grammatical features in the text; methods based on neural network models and machine learning require pre-extraction of implicit relationships between training texts before training the model, resulting in high complexity of text similarity determination methods. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for determining text similarity, which can reduce the complexity of the text similarity determination method and improve the accuracy of the text similarity determination results.

[0006] According to one aspect of the present invention, a method for determining text similarity is provided, comprising:

[0007] Multiple texts to be processed are obtained, and each text is segmented into words to obtain multiple segmentation results corresponding to each text.

[0008] Determine the computational natural language learning CoNLL format data corresponding to each of the word segmentation results, and construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts.

[0009] The CoNLL format data includes the corresponding word segmentation results and the grammatical relationships between them and other word segmentation results; the grammatical relationships include part-of-speech associations and semantic associations.

[0010] Based on the dependency syntax tree corresponding to each of the texts, a dependency matrix matching each of the texts is generated, and based on the dependency matrix, the target similarity between the multiple texts is determined.

[0011] Optionally, the word segmentation result includes words and the parts of speech corresponding to the words;

[0012] Determine the CoNLL format data corresponding to each of the aforementioned word segmentation results, including:

[0013] According to the order in which the words are arranged in the text, the segmentation results are sequentially input into the Chinese language processing tool HanLP.

[0014] The HanLP tool is used to determine the CoNLL format data corresponding to each of the word segmentation results.

[0015] Optionally, the CoNLL format data also includes the corresponding word segmentation results and the index relationship between them and other word segmentation results;

[0016] Based on the multiple CoNLL format data corresponding to each of the texts, construct the dependency syntax tree corresponding to each of the texts, including:

[0017] Based on the index and syntactic relationships between the word segmentation results in each of the CoNLL format data, a dependency syntax tree is constructed for each of the texts.

[0018] Optionally, based on the dependency syntax trees corresponding to each of the texts, a dependency matrix matching each of the texts is generated, including:

[0019] Obtain all word segmentation results corresponding to the multiple texts, and perform deduplication on the all word segmentation results to obtain multiple target word segmentation results;

[0020] Based on the multiple target word segmentation results, a target matrix is ​​generated that matches each of the texts respectively;

[0021] Based on the grammatical relationships between the word segmentation results in each dependency syntax tree, values ​​are assigned to each target matrix to obtain a dependency matrix that matches each text.

[0022] Optionally, before performing word segmentation on each of the texts, the method further includes:

[0023] Determine the edit distance between each of the texts, and based on the edit distance, determine the edit distance similarity between multiple texts;

[0024] Based on the dependency matrices, the target similarity between multiple texts is determined, including:

[0025] The target similarity between multiple texts is determined based on the edit distance similarity and each of the dependency matrices.

[0026] Optionally, based on the edit distance similarity and each of the dependency matrices, the target similarity between multiple texts is determined, including:

[0027] Determine the cosine similarity between each of the dependency matrices;

[0028] The edit distance similarity and the cosine similarity are weighted and summed according to a preset weight ratio to obtain the target similarity between multiple texts.

[0029] Optionally, based on the edit distance, determining the edit distance similarity between multiple texts includes:

[0030] Based on the edit distance and the text length corresponding to each text, the edit distance similarity between multiple texts is determined.

[0031] According to another aspect of the present invention, a text similarity determination apparatus is provided, the apparatus comprising:

[0032] The word segmentation module is used to acquire multiple texts to be processed, perform word segmentation on each text, and obtain multiple word segmentation results corresponding to each text.

[0033] The syntax tree construction module is used to determine the CoNLL format data corresponding to each of the word segmentation results, and to construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts.

[0034] The CoNLL format data includes the corresponding word segmentation results and the grammatical relationships between them and other word segmentation results; the grammatical relationships include part-of-speech associations and semantic associations.

[0035] The similarity determination module is used to generate a dependency matrix that matches each of the texts based on the dependency syntax tree corresponding to each text, and to determine the target similarity between multiple texts based on the dependency matrix.

[0036] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0037] At least one processor; and

[0038] A memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text similarity determination method according to any embodiment of the present invention.

[0040] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the text similarity determination method according to any embodiment of the present invention.

[0041] The technical solution provided by this invention involves acquiring multiple texts to be processed, performing word segmentation on each text to obtain multiple segmentation results corresponding to each text, determining the CoNLL format data corresponding to each segmentation result, constructing a dependency syntax tree for each text based on the multiple CoNLL format data corresponding to each text, generating a dependency matrix matching each text based on the dependency syntax tree, and determining the target similarity between multiple texts based on the dependency matrices. This approach reduces the complexity of text similarity determination methods and improves the accuracy of text similarity determination results. It preserves the original semantic expression order in the text, providing auxiliary information for the syntactic relationships between segmentation results; it reduces the time consumption of dependency syntax tree construction and improves the efficiency of dependency syntax tree construction; it allows the dependency matrix to contain rich semantic expression and syntactic structure information; by combining edit distance similarity with the dependency matrix, it can evaluate the similarity between texts at the string level and in terms of syntax; and it improves the accuracy of edit distance similarity determination results between texts and reduces the time consumption of target similarity determination methods.

[0042] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a text similarity determination method provided by an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of another text similarity determination method provided by an embodiment of the present invention;

[0046] Figure 3a This is a flowchart of another text similarity determination method provided by an embodiment of the present invention;

[0047] Figure 3bThis is a schematic diagram of a dependency syntax tree structure provided by an embodiment of the present invention;

[0048] Figure 3c This is a schematic diagram of another dependency syntax tree structure provided by an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of a text similarity determination device according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the text similarity determination method of this invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] Figure 1 This is a flowchart illustrating a text similarity determination method provided in an embodiment of the present invention. This embodiment is applicable to determining the similarity between multiple texts. The method can be executed by a text similarity determination device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0054] Step 110: Obtain multiple texts to be processed, perform word segmentation on each text, and obtain multiple word segmentation results corresponding to each text.

[0055] In this embodiment, the text to be processed can be text that requires similarity comparison. Specifically, for the retrieval system, the multiple texts to be processed can include text input by the user and text stored in the retrieval database. After obtaining multiple texts to be processed, the sentences in each text can be segmented to obtain multiple segmentation results corresponding to each text.

[0056] In one specific embodiment, sentences in each text can be segmented using a word segmentation tool. This word segmentation tool can be jieba, ltp, or ir, etc., and this embodiment is not limited to any particular tool.

[0057] Step 120: Determine the Conference on Computational Natural Language Learning (CoNLL) format data corresponding to each of the segmentation results, and construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts.

[0058] In this embodiment, after obtaining multiple word segmentation results corresponding to each text, a preset semantic recognition algorithm, such as Natural Language Processing (NLP) technology, can be used to process each word segmentation result to obtain CoNLL format data corresponding to each word segmentation result.

[0059] In a specific embodiment, a semantic recognition algorithm can be used to determine the implicit grammatical dependency rules between the segmentation results in the text based on the characteristics of Chinese grammar and the contextual relationships between different words, and to generate CoNLL format data corresponding to each segmentation result based on the implicit grammatical dependency rules between the segmentation results.

[0060] In this embodiment, the CoNLL format data includes the corresponding word segmentation result and the grammatical relationship between it and other word segmentation results. Specifically, the grammatical relationship may include the part-of-speech association relationship and the semantic association relationship between the corresponding word segmentation result and other word segmentation results.

[0061] In a specific embodiment, the CoNLL format data may also include lemmas or stems, parts of speech, and morphological feature lists corresponding to the word segmentation results.

[0062] In this step, optionally, a dependency syntax tree can be constructed for each text based on the syntactic relationships between the segmentation results in each CoNLL format data.

[0063] Step 130: Generate a dependency matrix that matches each of the texts based on the dependency syntax tree corresponding to each text, and determine the target similarity between multiple texts based on the dependency matrix.

[0064] In this embodiment, optionally, a dependency matrix matching each text can be generated based on the syntactic relationships between the segmentation results in each dependency syntax tree. Each data value in the dependency matrix represents the syntactic relationship between a specific segmentation result and other segmentation results. For example, if a specific segmentation result has a syntactic relationship with another segmentation result, the corresponding data value can be set to 1; conversely, if no syntactic relationship exists, the corresponding data value can be set to 0.

[0065] In one specific embodiment, optionally, after generating dependency matrices that match each text, the Euclidean distance or Mahalanobis distance between multiple dependency matrices can be calculated, and the target similarity between multiple texts can be determined based on the distance calculation results between multiple dependency matrices.

[0066] In this embodiment, by determining the CoNLL format data corresponding to each word segmentation result and constructing a dependency syntax tree corresponding to each text based on the multiple CoNLL format data corresponding to each text, the syntactic relationships between different texts can be fully extracted at the semantic level, providing auxiliary information for the text similarity determination method. This reduces the impact of different syntactic structures on the text similarity results and improves the accuracy of the text similarity determination results. Secondly, compared with the existing technology of extracting the implicit relationships between training texts before machine model training, this method can reduce the complexity of the text similarity determination method and improve the efficiency of text similarity determination.

[0067] The technical solution provided by this invention involves acquiring multiple texts to be processed, performing word segmentation on each text to obtain multiple word segmentation results corresponding to each text, determining the CoNLL format data corresponding to each word segmentation result, constructing a dependency syntax tree corresponding to each text based on the multiple CoNLL format data corresponding to each text, generating a dependency matrix matching each text based on the dependency syntax tree corresponding to each text, and determining the target similarity between multiple texts based on the dependency matrix. This approach can reduce the complexity of text similarity determination methods and improve the accuracy of text similarity determination results.

[0068] Figure 2 A flowchart of another text similarity determination method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0069] Step 210: Obtain multiple texts to be processed, perform word segmentation on each text, and obtain multiple word segmentation results corresponding to each text.

[0070] In this embodiment, optionally, word segmentation tools and a pre-built lexicon can be used to segment sentences in each text. The lexicon includes a large number of known words and their corresponding parts of speech (e.g., nouns, verbs, or adjectives). After word segmentation, the segmentation results include the words and their corresponding parts of speech.

[0071] Step 220: According to the order of the words in the text, input the word segmentation results into the Han Language Processing (HanLP) tool in sequence.

[0072] In this embodiment, optionally, multiple texts to be processed can be sorted, the target text to be processed can be determined sequentially according to the sorting results, and multiple word segmentation results corresponding to the target text can be obtained. Then, according to the order of words in each word segmentation result in the target text, each word segmentation result can be sequentially input into the HanLP tool.

[0073] Step 230: Using the HanLP tool, determine the CoNLL format data corresponding to each of the word segmentation results.

[0074] In this embodiment, optionally, the HanLP tool can be used to determine the CoNLL format data corresponding to each segmentation result based on the words and parts of speech in each segmentation result.

[0075] The advantage of this setup is that by inputting the segmentation results into the HanLP tool sequentially according to the order in which the words are arranged in the text, the CoNLL format data corresponding to the segmentation results can be quickly determined, improving the efficiency of text similarity determination. On the other hand, the original semantic expression order in the text can be preserved, providing auxiliary information for the grammatical relationship between the segmentation results, thereby improving the accuracy of the text similarity determination results.

[0076] In this embodiment, the CoNLL format data also includes the corresponding word segmentation results and the index relationship between them and other word segmentation results. Specifically, the index relationship may include a syntactic parent index between word segmentation results.

[0077] Step 240: Based on the index relationship and syntactic relationship between the word segmentation results in each of the CoNLL format data, construct the dependency syntax tree corresponding to each of the texts.

[0078] In this embodiment, optionally, other segmentation results associated with each segmentation result can be determined first based on the grammatical parent index of the segmentation results in each CoNLL format data. Then, based on the grammatical relations specified in each CoNLL format data, multiple segmentation results corresponding to each text can be connected to obtain the dependency syntax tree corresponding to each text.

[0079] The advantage of this setup is that by using the index and syntactic relationships between the segmentation results in each CoNLL format data, the dependency syntax tree corresponding to each text can be constructed, which can reduce the time spent building the dependency syntax tree and improve its construction efficiency.

[0080] Step 250: Obtain the full word segmentation results corresponding to the multiple texts, and perform deduplication processing on the full word segmentation results to obtain multiple target word segmentation results.

[0081] In this step, you can obtain all the word segmentation results for the entire text, then query the duplicate word segmentation results in the total word segmentation results, and remove the duplicate word segmentation results to obtain multiple target word segmentation results.

[0082] Step 260: Based on the multiple target word segmentation results, generate target matrices that match each of the texts respectively.

[0083] In this step, optionally, the total number of all target word segmentation results can be obtained, and then the number can be used as the row and column size to build a target matrix that matches each text separately.

[0084] Step 270: Assign values ​​to each target matrix according to the grammatical relationships between the word segmentation results in each dependency syntax tree to obtain the dependency matrix that matches each text respectively.

[0085] In this embodiment, the data values ​​in the dependency matrix are used to characterize the syntactic relationship between a certain target word segmentation result and other target word segmentation results among all target word segmentation results. For example, if a specific target word segmentation result has a syntactic relationship with another target word segmentation result, the corresponding data value can be set to 1; conversely, if there is no syntactic relationship, the corresponding data value can be set to 0.

[0086] The advantage of this setup is that by obtaining all the word segmentation results for all texts and establishing a dependency matrix of the same size as each text based on the word segmentation results, the dependency matrix can contain rich semantic expressions and syntactic structure information. By utilizing the rich semantic relationships in the dependency matrix, the accuracy of text similarity determination results can be improved.

[0087] Step 280: Determine the target similarity between multiple texts based on the dependency matrices described above.

[0088] The technical solution provided in this invention involves acquiring multiple texts to be processed, performing word segmentation on each text to obtain multiple word segmentation results, inputting each word segmentation result sequentially into the HanLP tool according to the order of words in the text, determining the CoNLL format data corresponding to each word segmentation result through the HanLP tool, constructing a dependency syntax tree corresponding to each text based on the index and syntactic relationships between the word segmentation results in each CoNLL format data, obtaining all word segmentation results corresponding to multiple texts, performing deduplication on all word segmentation results to obtain multiple target word segmentation results, generating target matrices that match each text based on the multiple target word segmentation results, assigning values ​​to each target matrix based on the syntactic relationships between the word segmentation results in each dependency syntax tree to obtain dependency matrices that match each text, and determining the target similarity between multiple texts based on each dependency matrix. This technical means can reduce the complexity of text similarity determination methods and improve the accuracy of text similarity determination results.

[0089] Figure 3a A flowchart of another text similarity determination method provided in an embodiment of the present invention is shown below. Figure 3a As shown, the method includes:

[0090] Step 310: Obtain multiple texts to be processed, determine the edit distance between each text, and determine the edit distance similarity between the multiple texts based on the edit distance.

[0091] Optionally, in this step, the edit distance between the texts can be determined using the Levenshtein Distance (LD) algorithm, and then a linear operation can be performed on the edit distances to obtain the edit distance similarity between the texts. The smaller the edit distance between the texts, the greater the corresponding edit distance similarity.

[0092] In one embodiment of this example, determining the edit distance similarity between multiple texts based on the edit distance includes: determining the edit distance similarity between multiple texts based on the edit distance and the text length corresponding to each text.

[0093] In a specific embodiment, assuming the text to be processed includes text A and text B, the text length of text A is A1, the text length of text B is B1, and the edit distance between text A and text B is C, then the edit distance similarity D between text A and text B can be calculated using the following formula:

[0094] D = 1 - C / max(A1, B1)

[0095] The advantage of this setting is that by obtaining the text length of the text to be processed, the accuracy of the edit distance similarity determination results between texts can be improved, thereby improving the accuracy of the subsequent target similarity determination results.

[0096] Step 320: Perform word segmentation on each of the texts to obtain multiple word segmentation results corresponding to each text.

[0097] Step 330: Determine the CoNLL format data corresponding to each of the word segmentation results, and construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts.

[0098] Step 340: Generate a dependency matrix that matches each of the texts based on the dependency syntax tree corresponding to each text.

[0099] Step 350: Determine the target similarity between multiple texts based on the edit distance similarity and each of the dependency matrices.

[0100] In this embodiment, optionally, the Euclidean distance or Mahalanobis distance between each dependency matrix can be calculated, and the distance calculation results between each dependency matrix and the edit distance similarity can be linearly calculated to obtain the final similarity (i.e., target similarity) between multiple texts.

[0101] The advantage of this setup is that, since edit distance similarity represents the degree of matching at the text string level, by combining edit distance similarity with the dependency matrix, the similarity between texts can be evaluated at both the string level and syntactic level, thereby improving the effectiveness and accuracy of text similarity determination results.

[0102] In one embodiment of this example, determining the target similarity between multiple texts based on the edit distance similarity and each of the dependency matrices includes: determining the cosine similarity between each of the dependency matrices; and weighting and summing the edit distance similarity and the cosine similarity according to a preset weight ratio to obtain the target similarity between the multiple texts.

[0103] In one specific embodiment, the cosine distance between each dependency matrix can be calculated, and then a linear operation can be performed on the cosine distance to obtain the cosine similarity. The smaller the cosine distance between the dependency matrices corresponding to two texts, the more similar the grammatical dependencies between the two texts are considered, i.e., the higher the cosine similarity.

[0104] The advantage of this setup is that by determining the cosine similarity between each dependency matrix, and then determining the target similarity between multiple texts based on the cosine similarity and edit distance similarity, the time consumption of the target similarity determination method can be reduced, and the accuracy of the target similarity can be improved.

[0105] The technical solution provided by this invention involves acquiring multiple texts to be processed, determining the edit distance between each text, determining the edit distance similarity between multiple texts based on the edit distance, performing word segmentation on each text to obtain multiple word segmentation results corresponding to each text, determining the CoNLL format data corresponding to each word segmentation result, constructing a dependency syntax tree corresponding to each text based on the multiple CoNLL format data corresponding to each text, generating a dependency matrix that matches each text based on the dependency syntax tree corresponding to each text, and determining the target similarity between multiple texts based on the edit distance similarity and each dependency matrix. This technical approach can reduce the complexity of text similarity determination methods and improve the accuracy of text similarity determination results.

[0106] Based on the above embodiments, this embodiment also provides a preferred implementation method, assuming that the acquired text to be processed includes text A and text B. Text A is "Mr. Xu ultimately decided to focus on painting eagles, squirrels, and sparrows.", and text B is "Mr. Xu determined to focus on painting eagles, squirrels, and sparrows.", then the text similarity determination method may include the following steps:

[0107] Step 1: Perform word segmentation on text A and text B respectively to obtain multiple word segmentation results.

[0108] In this step, after word segmentation, text A can be obtained as follows: Xu (NR) / Mr. (NN) / Finally (AD) / Decided (VV) / Le (AS) / Painted (VV) / Eagle (NN) / 、 (PU) / Squirrel (NN) / And (CC) / Sparrow (NN) / As (VV) / Main (AD) / Conquer (VV) / Of (DEC) / Direction (NN) / 。 (PU).

[0109] After word segmentation, text B yields the following results: Xu (NR) / Mr. (NN) / Confirm (VV) / Liao (AS) / Ba (BA) / Hua (VV) / Eagle (NN) / 、 (PU) / Squirrel (NN) / And (CC) / Sparrow (NN) / As (VV) / Main Attack (NN) / Target (NN) / 。 (PU).

[0110] Among them, each word segmentation result includes words and their corresponding word types, such as personal names (NR), work-related nouns (NN), adjectival adverbs (AD), other verbs (VV), auxiliary words (AS), punctuation marks (PU), coordinating conjunctions (CC), adverbs (DEC), onomatopoeic words (BA), etc.

[0111] Step 2: Input the multiple word segmentation results corresponding to text A and text B into the HanLP tool in sequence. Through the HanLP tool, determine the CoNLL format data corresponding to each word segmentation result. Taking text B as an example, the CoNLL format data corresponding to each word segmentation result in text B can be shown in Table 1. Among them, the CoNLL format data corresponding to each word segmentation result in Table 1 includes ID, word, the syntactic parent index of the word, and the syntactic relationship between the word and the associated word.

[0112] Specifically, as shown in Table 1, taking the word segmentation result "Xu (NR)" as an example, the ID of the CoNLL format data corresponding to this word segmentation result is 1, the word is "Xu", the syntactic parent index is 2 (that is, the associated word corresponding to this word is "Mr." with ID 2), and the syntactic relationship between this word and the associated word is compound noun (compound:nn).

[0113] Step 3: Construct the dependency syntax trees corresponding to text A and text B respectively according to the multiple CoNLL format data corresponding to text A and text B.

[0114] Table 1

[0115]

[0116]

[0117] In this step, taking text B as an example, the dependency syntax tree corresponding to text B can be constructed according to the index relationship and syntactic relationship between the word segmentation results in Table 1, as Figure 3b shown. Correspondingly, the dependency syntax tree corresponding to text A can be constructed in the same way as above, as Figure 3c shown.

[0118] Step 4: Obtain all the word segmentation results corresponding to Text A and Text B, perform duplicate removal on all the word segmentation results, and obtain the following target word segmentation results: "Xu", "Mr.", "determine", "have", "take", "draw", "eagle", ",", "squirrel", "and", "sparrow", "as", "the main", "target", ".", "Finally", "decide", "the main", "to overcome", "and", "direction", and then construct a target matrix according to the number of the above target word segmentation results. Finally, according to the dependency grammar trees corresponding to Text A and Text B respectively, assign values to the target matrix respectively to obtain the dependency matrices respectively matched by Text A and Text B.

[0119] Exemplarily, taking the dependency grammar tree corresponding to Text A (that is Figure 3c ) as an example, the dependency matrix matched by Text A can be shown in Table 2. Among them, the row and column dimensions of the dependency matrix are equal to the number (21) of the above target word segmentation results.

[0120] Step 5: Obtain the edit distance similarity between Text A and Text B, calculate the cosine similarity between the dependency matrix corresponding to Text A and the dependency matrix corresponding to Text B, and perform weighted summation on the edit distance similarity and the cosine similarity to obtain the final similarity between Text A and Text B.

[0121] Table 2

[0122]

[0123]

[0124] Through the above embodiments, the grammatical association relationship between Text A and Text B can be fully extracted at the semantic level, the influence of different grammatical structures on the text similarity result can be reduced, the determination efficiency of the text similarity and the accuracy of the similarity determination result can be improved.

[0125] Figure 4 The following is a schematic structural diagram of a text similarity determination device provided by an embodiment of the present invention. The device is applied to an electronic device, such as Figure 4 shown. The device includes: a word segmentation module 410, a grammar tree construction module 420, and a similarity determination module 430.

[0126] Among them, the word segmentation module 410 is used to obtain multiple texts to be processed, perform word segmentation on each of the texts, and obtain multiple word segmentation results respectively corresponding to each of the texts;

[0127] The syntax tree construction module 420 is used to determine the CoNLL format data corresponding to each of the word segmentation results, and construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts; the CoNLL format data includes the corresponding word segmentation result and the grammatical relationship between it and other word segmentation results; the grammatical relationship includes part-of-speech association relationship and semantic association relationship;

[0128] The similarity determination module 430 is used to generate a dependency matrix that matches each of the texts according to the dependency syntax tree corresponding to each of the texts, and to determine the target similarity between multiple texts according to the dependency matrix.

[0129] The technical solution provided by this invention involves acquiring multiple texts to be processed, performing word segmentation on each text to obtain multiple word segmentation results corresponding to each text, determining the CoNLL format data corresponding to each word segmentation result, constructing a dependency syntax tree corresponding to each text based on the multiple CoNLL format data corresponding to each text, generating a dependency matrix matching each text based on the dependency syntax tree corresponding to each text, and determining the target similarity between multiple texts based on the dependency matrix. This approach can reduce the complexity of text similarity determination methods and improve the accuracy of text similarity determination results.

[0130] Based on the above embodiments, the word segmentation results include words and the parts of speech corresponding to the words; the CoNLL format data also includes the corresponding word segmentation results and the index relationship between them and other word segmentation results.

[0131] The word segmentation module 410 includes:

[0132] An edit distance determination unit is used to determine the edit distance between each of the texts, and to determine the edit distance similarity between multiple texts based on the edit distance;

[0133] The edit distance similarity determination unit is used to determine the edit distance similarity between multiple texts based on the edit distance and the text length corresponding to each text.

[0134] Syntax tree building module 420 includes:

[0135] The word segmentation result input unit is used to input the word segmentation results into the HanLP tool in the order in which the words are arranged in the text.

[0136] The word segmentation result processing unit is used to determine the CoNLL format data corresponding to each word segmentation result through the HanLP tool;

[0137] The dependency syntax tree construction unit is used to construct a dependency syntax tree corresponding to each text based on the index relationship and syntactic relationship between the word segmentation results in each of the CoNLL format data.

[0138] The similarity determination module 430 includes:

[0139] The word segmentation and deduplication unit is used to obtain all word segmentation results corresponding to the multiple texts, perform deduplication on the all word segmentation results, and obtain multiple target word segmentation results;

[0140] The matrix generation unit is used to generate a target matrix that matches each of the texts based on the multiple target word segmentation results.

[0141] The dependency matrix generation unit is used to assign values ​​to each target matrix according to the grammatical relationship between the word segmentation results in each dependency syntax tree, so as to obtain the dependency matrix that matches each text respectively.

[0142] The target similarity determination unit is used to determine the target similarity between multiple texts based on the edit distance similarity and each of the dependency matrices.

[0143] A cosine similarity determination unit is used to determine the cosine similarity between each of the dependency matrices;

[0144] The weighted summation unit is used to sum the edit distance similarity and the cosine similarity according to a preset weight ratio to obtain the target similarity between multiple texts.

[0145] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in the embodiments of the present invention can be found in the methods provided in all the foregoing embodiments of the present invention.

[0146] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0147] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0148] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as text similarity determination methods.

[0150] In some embodiments, the text similarity determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the text similarity determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the text similarity determination method by any other suitable means (e.g., by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0156] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0157] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining text similarity, characterized in that, The method includes: Multiple texts to be processed are obtained, and each text is segmented into words to obtain multiple segmentation results corresponding to each text. Determine the computational natural language learning CoNLL format data corresponding to each of the word segmentation results, and construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts. The CoNLL format data includes the corresponding word segmentation results and the grammatical relationships between them and other word segmentation results; the grammatical relationships include part-of-speech associations and semantic associations. Based on the dependency syntax tree corresponding to each of the texts, a dependency matrix matching each of the texts is generated, and based on the dependency matrix, the target similarity between the multiple texts is determined.

2. The method according to claim 1, characterized in that, The word segmentation results include words and the parts of speech corresponding to those words; Determine the CoNLL format data corresponding to each of the aforementioned word segmentation results, including: According to the order in which the words are arranged in the text, the segmentation results are sequentially input into the Chinese language processing tool HanLP. The HanLP tool is used to determine the CoNLL format data corresponding to each of the word segmentation results.

3. The method according to claim 1, characterized in that, The CoNLL format data also includes the corresponding word segmentation results and the index relationship between them and other word segmentation results; Based on the multiple CoNLL format data corresponding to each of the texts, construct the dependency syntax tree corresponding to each of the texts, including: Based on the index and syntactic relationships between the word segmentation results in each of the CoNLL format data, a dependency syntax tree is constructed for each of the texts.

4. The method according to claim 1, characterized in that, Based on the dependency syntax trees corresponding to each of the texts, a dependency matrix matching each of the texts is generated, including: Obtain all word segmentation results corresponding to the multiple texts, and perform deduplication on the all word segmentation results to obtain multiple target word segmentation results; Based on the multiple target word segmentation results, a target matrix is ​​generated that matches each of the texts respectively; Based on the grammatical relationships between the word segmentation results in each dependency syntax tree, values ​​are assigned to each target matrix to obtain a dependency matrix that matches each text.

5. The method according to claim 1, characterized in that, Before performing word segmentation on each of the aforementioned texts, the following is also included: Determine the edit distance between each of the texts, and based on the edit distance, determine the edit distance similarity between multiple texts; Based on the dependency matrices, the target similarity between multiple texts is determined, including: The target similarity between multiple texts is determined based on the edit distance similarity and each of the dependency matrices.

6. The method according to claim 5, characterized in that, Based on the edit distance similarity and each of the dependency matrices, the target similarity between multiple texts is determined, including: Determine the cosine similarity between each of the dependency matrices; The edit distance similarity and the cosine similarity are weighted and summed according to a preset weight ratio to obtain the target similarity between multiple texts.

7. The method according to claim 5, characterized in that, Based on the edit distance, determine the edit distance similarity between multiple texts, including: Based on the edit distance and the text length corresponding to each text, the edit distance similarity between multiple texts is determined.

8. A text similarity determination device, characterized in that, The device includes: The word segmentation module is used to acquire multiple texts to be processed, perform word segmentation on each text, and obtain multiple word segmentation results corresponding to each text. The syntax tree construction module is used to determine the CoNLL format data corresponding to each of the word segmentation results, and to construct the dependency syntax tree corresponding to each of the texts based on the multiple CoNLL format data corresponding to each of the texts. The CoNLL format data includes the corresponding word segmentation results and the grammatical relationships between them and other word segmentation results; the grammatical relationships include part-of-speech associations and semantic associations. The similarity determination module is used to generate a dependency matrix that matches each of the texts based on the dependency syntax tree corresponding to each text, and to determine the target similarity between multiple texts based on the dependency matrix.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text similarity determination method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the text similarity determination method according to any one of claims 1-7.

Images