Text similarity recognition method, device, electronic device and readable storage medium

Abstract

The present invention provides a text similarity recognition method, device, electronic device and readable storage medium, the method comprising: obtaining a first text and a second text, and determining a common substring corresponding to the first text and the second text; determining the similarity contribution ratio of the common substring and the similarity contribution ratio of single characters in the first text and the second text except the common substring according to the word order of the first text and the second text respectively; determining the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character. The present invention not only takes into account the word order of the text, but also utilizes the contribution of the common substring and other single characters to the string similarity of the first text and the second text, which can improve the accuracy of the string similarity, so that the string similarity finally obtained meets the user's expectations.

Description

Technical Field

[0001] The present invention relates to the field of search technology, and in particular to a text similarity recognition method, device, electronic device and readable storage medium. Background Art

[0002] Text similarity analysis is a key research issue in both NLP and search fields, and is widely used in scenarios such as text mining, text clustering, search recall, and search ranking. Among them, string similarity is of great significance in text similarity analysis.

[0003] The more mature existing string similarity calculation methods are the Levenshtein Distance algorithm and the Jaccard similarity coefficient. The Jaccard similarity coefficient ignores the word order problem and is only applicable to text patterns that are not sensitive to word order. The Levenshtein Distance algorithm is based on adding, deleting, and modifying single characters at the end of the string, and only considers the absolute difference between strings, resulting in a large error between the calculated result and the actual similarity.

[0004] Therefore, how to provide a text similarity recognition method that is compatible with various text patterns and calculates similarity results with high accuracy is a problem that needs to be solved. Summary of the invention

[0005] One of the purposes of the present invention is to provide a similarity recognition method, device, electronic device and readable storage medium to be compatible with various text modes and improve the accuracy of string similarity.

[0006] In a first aspect, the present invention provides a text similarity recognition method, the method comprising: acquiring a first text and a second text, and determining a common substring corresponding to the first text and the second text; determining, according to the word order of the first text and the second text, a similarity contribution ratio of the common substring and a similarity contribution ratio of single characters in the first text and the second text other than the common substring; determining the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single characters.

[0007] In a second aspect, the present invention provides a text similarity recognition device, comprising: an acquisition module, used to acquire a first text and a second text, and determine a common substring corresponding to the first text and the second text; a determination module, used to determine, according to the word order of the first text and the second text, the similarity contribution ratio of the common substring and the similarity contribution ratio of single characters in the first text and the second text except the common substring; and an identification module, used to determine the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character.

[0008] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the method described in the first aspect.

[0009] In a fourth aspect, the present invention provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0010] The present invention provides a similarity recognition method, device, electronic device and readable storage medium. The method includes: obtaining a first text and a second text, and determining a common substring corresponding to the first text and the second text; determining the similarity contribution ratio of the common substring and the similarity contribution ratio of single characters other than the common substring in the first text and the second text respectively according to the word order of the first text and the second text; determining the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character. The present invention obtains the common substring corresponding to the first text and the second text, combines the word order of the first text and the second text, comprehensively considers the similarity contribution ratio corresponding to the common substring, and the similarity contribution ratio of other single characters other than the common substring to the first text and the second text, so that the final string similarity not only takes into account the text word order, but also utilizes the contribution of the common substring and other single characters to the string similarity of the first text and the second text, which can improve the accuracy of the string similarity, so that the final string similarity meets the user's expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A scene graph provided by an embodiment of the present invention;

[0013] Figure 2 A structural block diagram of an electronic device provided by an embodiment of the present invention;

[0014] Figure 3 A flowchart of a text similarity query method provided by an embodiment of the present invention;

[0015] Figure 4 A schematic diagram of the process of step S302 provided in an embodiment of the present invention;

[0016] Figure 5 A schematic diagram of an editing sequence of a common substring provided by an embodiment of the present invention;

[0017] Figure 6 A schematic diagram of a single-character editing sequence provided by the present invention;

[0018] Figure 7 A schematic diagram of the corresponding relationship between the number of obstruction areas and the obstruction value according to an embodiment of the present invention;

[0019] Figure 8 A flowchart of another text similarity recognition method provided by an embodiment of the present invention;

[0020] Fig. 9 A flowchart of another text similarity recognition method provided by an embodiment of the present invention;

[0021] Fig.10 A functional module diagram of a text similarity recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0025] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0026] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0027] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0028] Please refer to Figure 1 , Figure 1 A scenario graph is provided for an embodiment of the present invention, and the scenario may be, but is not limited to, text mining, text clustering, search recall, search ranking, and the like.

[0029] Reference Figure 1 As shown, the application environment involves a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected via a network. The user can access the search engine through the terminal 110, and the server 120 can be the server where the search engine is located. The terminal 110 or the server 120 can obtain the query content quary_1 input by the user and search for the quary_1. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

[0030] The query quary_1 entered by the user represents the question he wants to search for, which is a keyword related to the specific question after summary and refinement. After a search is initiated, the search engine recalls documents based on the quary_1 entered by the user and returns the search results page, which includes a list of documents similar to the query quary_1, such as "query_a", "query_b", "query_c" and "query_d", and returns the retrieved documents as a document collection to the user, who can choose to click on the document of interest to check it out to meet the query requirements.

[0031] In the above scenarios, text similarity analysis is a key research issue. Text similarity analysis technologies can be divided into two categories. The first category is similarity calculation in the form of supervised learning, self-supervised learning, and semi-supervised learning. The model method represented by Bert measures the similarity by the cosine similarity of the vector after Bert encoding, and can consider both the literal similarity and semantic similarity of the string. The second category is unsupervised similarity calculation. Common methods include the Jaccard similarity coefficient of the string and the edit distance of the string (the distance and similarity are inversely proportional).

[0032] It can be seen that although the first technical form can take into account both string similarity and semantic similarity, it is not very practical in some scenarios. For example, in the search scenario, if you want to determine the relevance between the query and the recalled content title, the query and the recalled content title should have a high string similarity, but because the query is usually short and the content title is long, the string similarity calculated by the first technical form is often low, which is inconsistent with the actual situation. Furthermore, considering that there may be search platforms that do not have excessively high requirements for semantic similarity, users have a greater demand for accurate measurement of string similarity. Therefore, studying string similarity is of great significance in text similarity analysis.

[0033] The more mature existing string similarity calculation methods are the Levenshtein Distance algorithm and the Jaccard similarity coefficient.

[0034] Among them, the Jaccard similarity coefficient uses the text intersection of two texts to calculate the similarity, ignoring the word order problem. It is only applicable to text patterns that are not sensitive to word order. For example, "retail wholesale" and "wholesale retail", the Jaccard similarity coefficient can be well compatible, but for scenarios that are very sensitive to word order, such as "1938" and "1839", the Jaccard similarity is 100%, but it is far from the actual similarity between the two texts.

[0035] The edit distance algorithm is based on adding, deleting and modifying single characters at the end of a string, and only considers the absolute difference between strings, resulting in a large error between the calculated result and the true similarity. For example, for the two texts "Shanghai Magic City" and "Magic City Shanghai", the true similarity should be very high, but the edit distance calculated by the edit distance algorithm is large, resulting in a very small string similarity and a large error between the calculated result and the true similarity.

[0036] It should be noted that the "real similarity" involved in the embodiments of the present invention refers to the similarity that meets the user's reasonable expectations. This reasonable expectation is the result summarized based on a large amount of experience. For example, the similarity expected by users for "Shanghai Magic City" and "Magic City Shanghai" should be very high. For "1938" and "1839", the similarity expected by users should be very low.

[0037] In order to solve the above-mentioned defects in string similarity calculation, how to provide a text similarity recognition method that is compatible with various text modes and calculates similarity results with high accuracy. An embodiment of the present invention provides a method for calculating string similarity. The method is compatible with various text modes, and the calculated string similarity meets the reasonable expectations of users.

[0038] First, an execution device of the text similarity recognition method provided by an embodiment of the present invention is introduced.

[0039] See also Figure 2 , Figure 2 A structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device may be Figure 1 The electronic device 200 may include: a memory 201, a processor 202, and a communication interface 203, wherein the memory 201, the processor 202, and the communication interface 203 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0040] The memory 201 can be used to store software programs and modules, such as the text similarity recognition device 400 instructions / modules provided in the embodiment of the present invention, which can be stored in the memory 201 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 200. The processor 202 executes the software programs and modules stored in the memory 201 to perform various functional applications and data processing. The communication interface 203 can be used to communicate signaling or data with other node devices.

[0041] Among them, the memory 201 can be but is not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0042] The processor 202 may be an integrated circuit chip with signal processing capability. The processor 202 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0043] Understandably, Figure 2 The structure shown is for illustration only. The electronic device 200 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 The components shown may be implemented in hardware, software or a combination thereof.

[0044] See also Figure 3 , Figure 3 A flowchart of a text similarity query method provided in an embodiment of the present invention is provided. The text similarity (or similarity) referred to in the present invention only refers to string similarity, and semantic similarity is not within the scope of consideration of the embodiment of the present invention. The method can be Figure 2 The electronic device shown is an execution subject, and the method includes:

[0045] S301, obtaining a first text and a second text, and determining a common substring corresponding to the first text and the second text.

[0046] In this embodiment, the common substring represents the same character component in the first text and the second text, and the text length of the same character component is greater than or equal to 2.

[0047] For example, the first text is "Shanghai is the Magic City, and it was called the "Ten Miles of Foreign Lands" in the old days", and the second text is "The Magic City Shanghai, which was called the "Ten Miles of Foreign Lands" in the old days", then the common substrings corresponding to the first text and the second text are: Shanghai, Magic City, the old days, and the "Ten Miles of Foreign Lands"; for another example, the first text and the second text are "Hema Fresh" and "Hema Fresh" respectively, then the common substring is Hema.

[0048] In an optional implementation, N-Gram may be used to segment the first text and the second text to obtain their respective sentence components (Gram), and then the Grams corresponding to the first text and the second text are compared for component consistency, thereby obtaining a common substring.

[0049] For example: the first text is "community gate" and the second text is "gate east". For example: using 3-Gram to cut the first text "community gate", the obtained Gram is: community big, community, small, district gate, district big, district, gate, big, gate; using 3Gram to cut the second text "gate east", the obtained Gram is: gate, big, gate east, gate, east. After comparing the component consistency, the common substring is: gate.

[0050] S302: Determine, according to the word order of the first text and the second text, the similarity contribution ratio of the common substring and the similarity contribution ratio of the single characters in the first text and the second text except the common substring.

[0051] In the embodiment of the present invention, word order represents the relative position relationship between each character or character string in the text, and word order has a direct impact on the semantics of the text. For example, when comparing the text "1983" and the text "1938", the two texts have the same characters but very different semantics. The embodiment of the present invention considers the impact of word order on similarity before calculating the string similarity, which can greatly improve the accuracy of string similarity.

[0052] In this embodiment, the above-mentioned common substring similarity contribution ratio represents the contribution ratio of the common substring itself and the word order difference information of the common substring in the first text and the second text to the string similarity between the first text and the second text. Similarly, the single character similarity contribution ratio represents the contribution ratio of the single character string itself and the word order difference information of the single character in the first text and the second text to the string similarity between the first text and the second text.

[0053] It is understandable that if the actual string similarity of two texts is very high, they should contain more identical character components, and the contribution ratio of other single characters to the string similarity except the identical character components should be much smaller than the contribution ratio of the identical character components.

[0054] That is to say, for two texts, if the contribution ratio of the same character components is high, then the string similarity of the two texts should theoretically be high. On the contrary, if the two texts do not contain the same character components or contain very few, then the string similarity of the two texts is correspondingly low.

[0055] By calculating the contribution ratio of the common substring and other characters except the common substring to the string similarity, the final string similarity is made consistent with the actual situation.

[0056] S303, determining the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character.

[0057] In this embodiment, the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character are added together to obtain the string similarity between the first text and the second text.

[0058] According to the text similarity recognition method provided by the embodiment of the present invention, by obtaining the common substring corresponding to the first text and the second text, combined with the word order of the first text and the second text, the similarity contribution ratio corresponding to the common substring and the similarity contribution ratio of other single characters except the common substring to the first text and the second text are comprehensively considered, so that the final string similarity not only takes into account the text word order, but also utilizes the contribution of the common substring and other single characters to the string similarity of the first text and the second text, which can improve the accuracy of the string similarity, so that the final string similarity meets the user's expectations.

[0059] The above text similarity recognition method can be applied to various scenarios such as text mining, text clustering, search recall, search ranking, etc. The following takes the search ranking scenario as an example to introduce the above method in detail.

[0060] The server 120 obtains the query sentence (quary) sent from the terminal 110, that is, the first text, and retrieves the target rewritten sentence (tagertquary) corresponding to the quary from the quary library, that is, the second text. The tagertquary and the quary meet the preset similarity threshold. Generally speaking, one quary may correspond to multiple tagsertquaries, so these multiple tagsertquaries can all be used as the second text.

[0061] For each tagertquary, the server 120 uses N-Gram to cut the quary and the tagertquary to obtain their corresponding character components.

[0062] The server 120 may perform a component consistency comparison on the character components corresponding to quary and tagertquary, and determine the same character components as the common substrings corresponding to uary and tagertquary.

[0063] Then, the server 120 obtains the word order information of the common substring in quary and tagertquary respectively, calculates the similarity contribution ratio corresponding to the common substring, and then obtains the word order information of the single character except the common substring, and calculates the similarity contribution ratio corresponding to the single character.

[0064] For example, the first example is quary is "Shanghai is the Magic City, it was called the "Ten Miles of Foreign Lands" in the old days", tagertquary is "The Magic City Shanghai, which was called the "Ten Miles of Foreign Lands" in the old days", and the common substrings are "Shanghai", "Magic City", "Old Times", and "Ten Miles of Foreign Lands". If the above common substrings are replaced by ABCD respectively, quary can be simplified as "A is B, C calls it D", and tagertquary can be simplified as "C is called D's BA". Then in quary, the word order of the common substring is "A, B, C, D", and in tagertquary, the word order of the common substring is "C, D, B, A". Similarly, the word order information in single-character quary and tagertquary can be obtained.

[0065] Finally, the server 120 sums up the similarity contribution ratios corresponding to the common substrings and the similarity contribution ratios corresponding to the single characters, and thus the string similarity between the query statement and any target rewritten statement can be obtained. In combination with the string similarity corresponding to each target rewritten statement, an arrangement strategy for the documents corresponding to all target rewritten statements can be specified, and the arrangement strategy can be sent to the terminal 110, which then displays the documents corresponding to the target rewritten statements in sequence according to the arrangement strategy. It can be considered that the target rewritten statement with the greater string similarity with the quary will be ranked higher in the sorting result.

[0066] The following embodiments of the present invention will describe in detail how to calculate the similarity contribution ratio of the common substring and the similarity contribution ratio of the single characters in the first text and the second text except the common substring.

[0067] See also Figure 4 , Figure 4 The flowchart of step S302 provided in the embodiment of the present invention is as follows: Figure 4 As shown, step S302 may include the following sub-steps:

[0068] Sub-step S302-1: Determine the minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order according to the first word order of the common substring in the first text and the second word order of the common substring in the second text.

[0069] In the embodiment of the present invention, the first word order represents the relative position relationship of each common substring in the first text, and similarly, the second word order represents the relative position relationship of each common substring in the second text. The above-mentioned minimum word order adjustment number represents the minimum number of editing operations required to adjust the relative position relationship of the common substring in the first text to the relative position relationship of the common substring in the second text, which can also be called the word order adjustment cost, and the editing operation can be addition, deletion and modification.

[0070] For example, continuing to refer to the first example, the first word order is "A, B, C, D", and the second word order is "C, D, B, A", then the minimum number of moves or exchanges required to convert ABCD into CDBA is the minimum number of word order adjustments.

[0071] In an optional implementation, the method for determining the minimum number of word order adjustments of a common substring may refer to step:

[0072] a1, obtain a first character string corresponding to the first text and a second character string corresponding to the second text; wherein the first character sequence includes a common substring and a first word order; and the second character string includes a common substring and a second word order.

[0073] In this embodiment, the first string is a string composed of the editing characters corresponding to the common substring and the relative position relationship of the common substring in the first text, and the second string is a string composed of the common substring and the relative position relationship of the common substring in the second text. The common substring is replaced by the editing characters, which can facilitate the subsequent rapid acquisition of the editing sequence and improve the editing efficiency.

[0074] In an optional embodiment, the present invention is implemented in the process of obtaining the first string and the second string, firstly replacing the common substring with a special character, and then combining the special characters representing the common substring according to the relative position relationship between the common substrings, so as to obtain the above-mentioned first string and second string, for example, the first string is ABCD in the above-mentioned first example, and the second string is CDBA in the above-mentioned first example, wherein A, B, C, and D are the editing characters corresponding to the common substring respectively.

[0075] In this way, the subsequent word order adjustment process not only improves the editing operation efficiency, but also does not need to split the same sentence components, which can save editing operation steps and avoid the problem of large errors in calculation results based on single-character editing like the existing edit distance algorithm.

[0076] For example, assuming that the first text and the second text are actually "Hema Fresh" and "Fresh Hema", since the existing edit distance algorithm has the limitation of only adding, deleting and modifying at the end of each step, at least 4 exchanges are required to convert "Hema Fresh" into "Fresh Hema". However, using the above method of the embodiment of the present invention, the first text is converted into the first string EF, and the second text is converted into the second string FE, so only one conversion is required.

[0077] a2, editing and converting the common substring in the first string until the converted first string is consistent with the second string, thereby obtaining an edit sequence of the common substring;

[0078] Among them, the editing sequence is the optimal editing sequence obtained by editing and converting each character in the string using different editing modes through a dynamic programming algorithm. It is used to maintain multiple editing modes corresponding to a common substring. These multiple editing modes have a sequential execution order. The editing modes include: insert mode (ins), replace mode (sub) and delete mode (del).

[0079] For example, referring to the first example above, the first character string and the second character string are ABCD and CDBA respectively. Then the editing sequence for converting "ABCD" into "CDBA" can be referred to as Figure 5 As shown, Figure 5 A schematic diagram of an editing sequence of a common substring provided by an embodiment of the present invention.

[0080] like Figure 5 As shown, the edit sequence (editSequence) records the process of converting "ABCD" to "CDBA", wherein "sub"->"A->C" represents an editing mode, that is, replacing A with C.

[0081] a3, from the editing sequence, count the editing pattern pairs with symmetric relationship, and use the number of editing pattern pairs as the minimum number of word order adjustments.

[0082] In the embodiment of the present invention, two opposite editing conversions are performed on the same or a pair of common substrings, then the two editing modes corresponding to the two opposite editing conversions can form an editing mode pair, that is, after a certain two characters are replaced once, they must be replaced again in the reverse direction, after a certain character is inserted, the character must be deleted, and after a certain character is deleted, the character must be inserted.

[0083] For example, continue to refer to Figure 5 "sub-AC" and "sub-CA" can form an edit mode pair with a symmetrical relationship, representing that A is replaced by C and then C is replaced by A. For another example, "ins-D" and "del-D" can also form an edit mode pair with a symmetrical relationship, that is, D is first inserted and then deleted.

[0084] It is understandable that when the order of characters in a string is randomly shuffled to form a new string, the editing sequence from the original string to the new string will show this symmetry rule. If it is assumed that this symmetry rule is not satisfied, it will violate the premise that the new string is obtained by randomly adjusting the order of the original string. Therefore, as long as the total number of occurrences of this symmetry phenomenon, that is, the number of pairs of editing modes with a symmetric relationship, is counted, the minimum number of word order adjustments can be obtained.

[0085] It should be noted that in some scenarios, the replace edit mode may not appear symmetrically in the edit sequence, but the replace edit mode can be equivalently split into a delete edit mode and an insert edit mode with a symmetrical relationship, which can still be regarded as a symmetrical appearance.

[0086] At this point, through the above steps a1 to a3, the minimum number of word order adjustments after adjusting the word order of the common substring in the first text and the second text can be obtained. Then, on this basis, the contribution value of the common substring can be obtained by combining the length information of the common substring and the first text.

[0087] The following describes the processing of a single character. Please continue to refer to sub-step S302-2.

[0088] Sub-step S302-2, determining the character conversion barrier value and the number of identical characters between the first character and the second character according to the third word order of the first character in the first text and the fourth word order of the second character in the second text.

[0089] In the embodiments of the present invention, for the convenience of description, each single character in the first text is referred to as the first single character, each single character in the second text is referred to as the second single character, the third word order represents the relative position relationship between the first single characters in the first text except for the common substring, and similarly, the fourth word order represents the relative position relationship between the second single characters in the second text except for the common substring.

[0090] The above-mentioned identical single characters refer to single characters with the same character but without the need for editing conversion. Simply understood, they are single characters with the same character and the same word order. For example, in the text "The head of the kitten is really round" and the text "The whiskers of the kitten are really long", the word "of" has the same word order in both texts and does not require editing conversion, so it can be an identical single character. However, although the character "really" is the same, its word order is different and requires editing conversion, so it is not an identical single character.

[0091] It can be understood that for two texts, if there is no need to adjust the word order between two single characters, it means that these two single characters are close in position in the two texts. Only in this case does the contribution of the single character to the string similarity be positive. Otherwise, these two single characters make no contribution to the string similarity. By counting the number of identical single characters, the positive contribution degree of the single string can be determined.

[0092] In an optional embodiment, the above sub-step S302-1 can be executed according to the following steps:

[0093] b1. Obtain the third string corresponding to the first text and the fourth string corresponding to the second text; wherein, the third character sequence includes the first single character and the third word order; the fourth string includes the second single character and the fourth word order.

[0094] For the convenience of subsequent editing and conversion of single characters, before obtaining the third string and the fourth string, the punctuation symbols in the first text and the second text can be converted into symbols in the first preset format. For example, use "_" to replace punctuation symbols. Similarly, in order to avoid the influence of the common substring on the single character conversion result, the common substrings in the first text and the second text can also be converted into symbols in the second preset format. For example, "#" can be used to replace the common substring. After the above preprocessing, the third string and the fourth string are obtained.

[0095] It can be understood that the third string is the first text after preprocessing the common substring and punctuation characters in the first text. Similarly, the fourth string is the second text after preprocessing the common substring and punctuation characters in the second text.

[0096] b2, edit and convert the first single character in the third character string until the converted third character string is consistent with the fourth character string, thereby obtaining an edit sequence; wherein the edit sequence is used to maintain multiple edit modes for converting the first single character into the second single character.

[0097] For example, referring to the first example, the first text is "Shanghai is the Magic City, formerly known as the "Ten Miles of Foreign Lands", and the second text is "The Magic City Shanghai, formerly known as the "Ten Miles of Foreign Lands", and all common substrings are replaced with the special character "#", then the third string is "# is #_# called #", and the fourth string is "# is called #'s ##".

[0098] See also Figure 6 , Figure 6 A schematic diagram of a single-character editing sequence provided by the present invention, Figure 6 This is the editing sequence obtained by taking the two texts in the first example as an example. It can be seen that "# is #, # calls it #" is Figure 6 Multiple editing modes in the , you can convert "# is #_# calls it #" into "# is called #'s ##".

[0099] b3, from the edit sequence, determine the character conversion barrier value and the number of identical characters according to the edit mode type between the first character and the second character.

[0100] In an optional implementation manner, the above b3 can be performed according to the following steps:

[0101] b3-1, from the editing sequence, count the total number of obstruction areas corresponding to non-empty editing patterns.

[0102] The non-empty editing mode represents any one of the following editing modes: replace mode, delete mode, and insert mode.

[0103] Continuing with 6, the replacement mode, deletion mode, and insertion mode can be regarded as obstacles. The more obstacles appear in succession, the greater the obstacle value of this obstacle area. Figure 6 Including the area before the first character and after the last character, there are 4 obstruction areas in total.

[0104] It should be noted that the obstruction area and the obstruction value are both modeling tools for determining the string similarity calculation model in the embodiment of the present invention and have no other meanings.

[0105] b3-2, obtaining a statistical sequence according to the total number of the obstruction areas, and calculating the character conversion obstruction value based on a preset corresponding relationship between the total number of the obstruction areas and the obstruction value.

[0106] In the embodiment of the present invention, the statistical sequence is obtained based on the total number of obstruction areas in step b3-1, and the sequence value in the statistical sequence is an integer value between 1 and the total number of obstruction areas (including 1 and the total number of obstruction areas). Figure 6 If there are 4 obstruction areas, then the statistical sequence is [1,2,3,4], and each sequence value in the statistical sequence can be regarded as the number of obstruction areas.

[0107] In an optional implementation, the corresponding relationship between the number of obstruction areas and the obstruction value can be as follows: Figure 7 As shown, Figure 7 FIG. 1 is a schematic diagram of a corresponding relationship between the number of obstruction areas and the obstruction value according to an embodiment of the present invention. Figure 7 As shown, cnt represents the total number of obstruction areas, value represents the obstruction value, when cnt is less than or equal to 2, the corresponding obstruction value is 1, when cnt is greater than 2 and less than or equal to 4, the corresponding obstruction value is 1, when cnt is greater than 4 and less than or equal to 6, the corresponding obstruction value is 1.5, and when cnt is greater than 6, the corresponding obstruction value is 2.

[0108] according to Figure 7 The character conversion obstacle value can be obtained by summing the number of obstacle areas in the statistical sequence and the obstacle values ​​corresponding to the number of obstacle areas. In an optional implementation, the character conversion obstacle value can be obtained according to the following relationship:

[0109]

[0110] Wherein, obstructValue represents the character conversion obstruction value; spanList is a temporary table used to record the obstruction value corresponding to the number of obstruction areas. For example, spanList[0] records the obstruction value 0.5 corresponding to the number of 1 obstruction area, and so on; i represents the i-th sequence value in the statistical sequence;

[0111] b3-3, from the edit sequence, the number of edit modes other than the target edit mode is taken as the number of identical single characters, wherein the target edit mode is a non-empty edit mode and an edit mode with a common substring.

[0112] It should be noted that the positive contribution of a single character must be the same as that of the edit sequence, which means that the matched single characters must be in a similar position in the original text, otherwise there is no contribution. Therefore, we consider counting the number of identical single characters in the edit sequence instead of the original sequence single characters. Figure 6 The number of identical single characters (singleCharNum) in is 0.

[0113] Sub-step S302-3, based on the minimum number of word order adjustments, the total number of characters in the common substring, and the length information of the first text, determine the similarity contribution ratio of the common substring, and based on the character conversion barrier value, the number of identical single characters, and the length information of the first text, determine the similarity contribution ratio of the single character.

[0114] In the embodiment of the present invention, the similarity contribution ratio of the common substring and the minimum number of word order adjustments, as well as the length information of the common substring and the first text may satisfy the following relationship:

[0115]

[0116] Among them, similarity(gram) represents the similarity contribution ratio of the common substring; gramContrib represents the total number of characters in the common substring; queryLength represents the length information of the first text; minimumOrderAdjustmentCost represents the minimum number of word order adjustments; and p represents the constant coefficient.

[0117] Similarly, the similarity contribution ratio of a single character can be obtained according to the following relationship:

[0118]

[0119] Among them, similarity(gram) represents the similarity contribution ratio of a single character; singleCharNum represents the number of identical single characters; queryLength represents the length information of the first text; obstructValue represents the character conversion obstacle value; and p represents the constant coefficient.

[0120] Then, through the above two equations, we can finally get the string similarity between the first text and the second text, which can be expressed as:

[0121]

[0122] In an optional implementation, after obtaining the string similarity between two texts, the similarity between the two texts can also be interpreted, see Figure 8 , Figure 8 A flowchart of another text similarity recognition method provided by an embodiment of the present invention, the method may also include:

[0123] S304, obtaining a plurality of preset similarity measurement intervals and a similarity label corresponding to each similarity measurement interval.

[0124] S305: Match the string similarity with multiple similarity measurement intervals, and assign similarity tags corresponding to the matched similarity measurement intervals to the first text and the second text.

[0125] In an embodiment of the present invention, similarity measurement intervals and similarity labels can be defined according to actual needs. In the definition process, the string similarity calculated by the embodiment of the present invention and the user can be combined to divide the intervals. In an optional implementation, the above-mentioned multiple similarity measurement intervals can be 0-0.2, 0.2-0.35, 0.35-0.5, 0.5-0.65, 0.65-0.8, 0.8-1, and the similarity labels corresponding to these similarity measurement intervals are: very dissimilar, dissimilar, somewhat similar, similar, very similar, and almost synonymous.

[0126] For ease of understanding, the following lists some string similarities between the first text and the second text, as well as the matched similarity measurement intervals and similarity labels:

[0127] ! Jiang Xiaoya—> Jiang Xiaoya: 0.41666666666666667, 0.35-0.5, somewhat similar;

[0128] Sword Emperor -> Sword Dao Emperor: 0.5833333333333334, 0.5-0.65, similar;

[0129] Where did the money go—>Where did the money go: 0.86666666666666667, 0.8-1, almost synonymous.

[0130] Shanghai is the Magic City, which was called the "Ten Miles of Foreign Lands" in the old days—>The Magic City Shanghai, which was called the "Ten Miles of Foreign Lands" in the old days: 0.52991452991453, 0.5-0.65, similar.

[0131] In an optional implementation, when there are multiple second texts, the arrangement order between the multiple second texts can also be determined based on the obtained string similarity, and the arrangement order can be used as a reference for how to display on the display interface of the terminal, see Fig. 9 , Fig. 9 A flowchart of another text similarity recognition method provided by an embodiment of the present invention, the method may also include:

[0132] S306, when there are multiple second texts, respectively obtain the string similarity between each second text and the first text;

[0133] S307: Determine a similarity ranking strategy for each second text based on the string similarity between each second text and the first text.

[0134] Based on the same inventive concept, Fig.10 A functional module diagram of a text similarity recognition device provided by an embodiment of the present invention, referring to Fig.10 As shown, the text similarity recognition device 400 includes: an acquisition module 410 , a determination module 420 and a recognition module 430 .

[0135] An acquisition module 410 is used to acquire a first text and a second text, and determine a common substring corresponding to the first text and the second text;

[0136] A determination module 420, configured to determine, according to the word order of the first text and the second text, a similarity contribution ratio of a common substring and a similarity contribution ratio of single characters other than the common substring in the first text and the second text;

[0137] The recognition module 430 is used to determine the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character.

[0138] It is understandable that the acquisition module 410, the determination module 420 and the identification module 430 can be executed in coordination. Figure 3 Each step in the process is performed to achieve the corresponding technical effect.

[0139] In an optional implementation, the determination module 420 is specifically used to: determine the minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order according to the first word order of the common substring in the first text and the second word order in the second text; determine the character conversion barrier value and the number of identical characters between the first character in the first text and the second character in the second text according to the third word order of the first character in the first text and the fourth word order of the second character in the second text; determine the similarity contribution ratio of the common substring based on the minimum number of word order adjustments and the length information of the common substring and the first text, and determine the similarity contribution ratio of the single characters based on the character conversion barrier value, the number of identical characters and the length information of the first text.

[0140] In an optional implementation, the determination module 420 is specifically used to: determine the minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order according to the first word order of the common substring in the first text and the second word order in the second text, including: obtaining a first character string corresponding to the first text and a second character string corresponding to the second text; wherein the first character sequence includes the common substring and the first word order; the second character string includes the common substring and the second word order; editing and converting the common substring in the first character string until the converted first character string is consistent with the second character string, and obtaining an editing sequence of the common substring; wherein the editing sequence is used to maintain multiple editing modes corresponding to the common substring; from the editing sequence, counting editing mode pairs with a symmetrical relationship, and taking the number of editing mode pairs as the minimum number of word order adjustments.

[0141] In an optional implementation, the determination module 420 is specifically used to: determine the character conversion barrier value and the number of identical characters between the first character and the second character according to the third word order of the first character in the first text and the fourth word order of the second character in the second text; including: obtaining a third character string corresponding to the first text and a fourth character string corresponding to the second text; wherein the third character sequence includes the first character and the third word order; the fourth character string includes the second character and the fourth word order; editing and converting the first character in the third character string until the converted third character string is consistent with the fourth character string, thereby obtaining an editing sequence; wherein the editing sequence is used to maintain multiple editing modes for converting the first character into the second character; from the editing sequence, determining the character conversion barrier value and the number of identical characters according to the type of editing mode between the first character and the second character.

[0142] In an optional implementation, the determination module 420 is specifically used to: count the total number of obstacle areas corresponding to non-empty editing modes from the editing sequence; wherein the non-empty editing mode represents any one of the following editing modes: replacement mode, deletion mode and insertion mode; obtain a statistical sequence according to the total number of obstacle areas, and calculate the character conversion obstacle value based on the corresponding relationship between the preset total number of obstacle areas and the obstacle value; from the editing sequence, take the number of editing modes other than the target editing mode as the number of identical single characters, wherein the target editing mode is a non-empty editing mode and an editing mode with a common substring.

[0143] In an optional implementation, the text similarity recognition device 400 further includes a configuration module, and the acquisition module 410 is further used to obtain a plurality of preset similarity measurement intervals and a similarity label corresponding to each similarity measurement interval; the configuration module is used to match the string similarity with the plurality of similarity measurement intervals, and configure the similarity label corresponding to the matched similarity measurement interval to the first text and the second text;

[0144] In an optional implementation, the acquisition module 410 is also used to obtain the string similarity between each second text and the first text when there are multiple second texts; the determination module 420 is also used to determine the similarity arrangement strategy of each second text based on the string similarity between each second text and the first text.

[0145] The embodiment of the present invention further provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the query statement library construction method as described in any one of the above embodiments is implemented. The computer readable storage medium can be, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a magnetic disk, or an optical disk.

[0146] It should be understood that the device and method disclosed in the present invention can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0148] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes. It should be noted that in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0149] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

Claims

1. A text similarity recognition method, characterized in that: The method comprises: Acquire a first text and a second text, and determine a common substring corresponding to the first text and the second text; Determining, according to the word order of the first text and the second text, the similarity contribution ratio of the common substring and the similarity contribution ratio of single characters in the first text and the second text except the common substring; Determining the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character; Determining, according to the word order of the first text and the second text, the similarity contribution ratio of the common substring and the similarity contribution ratio of single characters in the first text and the second text except the common substring, respectively, includes: Determining, according to a first word order of the common substring in the first text and a second word order of the common substring in the second text, a minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order; Determine a character conversion barrier value and the number of identical characters between the first character and the second character according to a third word order of a first character in the first text and a fourth word order of a second character in the second text; Based on the minimum number of word order adjustments and the length information of the common substring and the first text, the similarity contribution ratio of the common substring is determined, and based on the character conversion barrier value, the number of identical single characters and the length information of the first text, the similarity contribution ratio of the single character is determined.

2. The text similarity recognition method according to claim 1, characterized in that: Determining, according to the first word order of the common substring in the first text and the second word order of the common substring in the second text, a minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order, includes: Obtain a first string corresponding to the first text and a second string corresponding to the second text; wherein the first string includes the common substring and the first word order; and the second string includes the common substring and the second word order; Editing and converting the common substring in the first string until the converted first string is consistent with the second string, and obtaining an edit sequence of the common substring; wherein the edit sequence is used to maintain multiple edit modes corresponding to the common substring; From the editing sequence, edit mode pairs having a symmetrical relationship are counted, and the number of edit mode pairs is used as the minimum word order adjustment times.

3. The text similarity recognition method according to claim 1, characterized in that: Determining a character conversion barrier value and a number of identical characters between the first character and the second character according to a third word order of a first character in the first text and a fourth word order of a second character in the second text; comprising: Obtain a third character string corresponding to the first text and a fourth character string corresponding to the second text; wherein the third character string includes the first single character and the third word order; and the fourth character string includes the second single character and the fourth word order; Editing and converting the first single character in the third character string until the converted third character string is consistent with the fourth character string, thereby obtaining an editing sequence; wherein the editing sequence is used to maintain multiple editing modes for converting the first single character into the second single character; The character conversion barrier value and the number of identical single characters are determined from the editing sequence according to the type of editing mode between the first single character and the second single character.

4. The text similarity recognition method according to claim 3, characterized in that: Determining the character conversion barrier value and the number of identical single characters from the editing sequence according to the type of editing mode between the first single character and the second single character comprises: From the editing sequence, counting the total number of obstruction areas corresponding to non-empty editing modes; wherein the non-empty editing mode represents any one of the following editing modes: replacement mode, deletion mode and insertion mode; According to the total number of the obstruction areas, a statistical sequence is obtained, and based on a preset corresponding relationship between the total number of the obstruction areas and the obstruction value, the character conversion obstruction value is calculated; From the editing sequence, the number of editing modes other than the target editing mode is used as the number of identical single characters, wherein the target editing mode is the non-empty editing mode and the editing mode having a common substring.

5. The text similarity recognition method according to claim 1, characterized in that: The method further comprises: Obtaining a plurality of preset similarity measurement intervals and a similarity label corresponding to each similarity measurement interval; The string similarity is matched with the multiple similarity measurement intervals, and similarity tags corresponding to the matched similarity measurement intervals are configured for the first text and the second text.

6. The text similarity recognition method according to claim 1, characterized in that: The method further comprises: When there are multiple second texts, respectively obtain the string similarity between each second text and the first text; Based on the string similarity between each second text and the first text, a similarity ranking strategy for each second text is determined.

7. A text similarity recognition device, characterized in that: include: An acquisition module, configured to acquire a first text and a second text, and determine a common substring corresponding to the first text and the second text; A determination module, configured to determine, according to the word order of the first text and the second text, a similarity contribution ratio of the common substring and a similarity contribution ratio of single characters in the first text and the second text except the common substring; A recognition module, configured to determine the string similarity between the first text and the second text according to the similarity contribution ratio of the common substring and the similarity contribution ratio of the single character; The determining module is specifically configured to determine, based on the first word order of the common substring in the first text and the second word order of the common substring in the second text, the minimum number of word order adjustments of the common substring when adjusting the first word order of the common substring to the second word order; According to the third word order of the first character in the first text and the fourth word order of the second character in the second text, determine the character conversion barrier value and the number of identical characters between the first character and the second character; based on the minimum word order adjustment times and the length information of the common substring and the first text, determine the similarity contribution ratio of the common substring, and based on the character conversion barrier value, the number of identical characters and the length information of the first text, determine the similarity contribution ratio of the single characters.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images