Text comparison method, device, electronic device and computer-readable storage medium

Abstract

The present application provides a text comparison method, device, electronic device and computer-readable storage medium, the method comprising: traversing a first text, obtaining a fingerprint value of a first fingerprint of each string in the first text; traversing a second text, obtaining a fingerprint value of a second fingerprint of each string in the second text; comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint, and obtaining the similarity between the second text and the first text. The scheme of the embodiment of the present application has a time complexity of approximately O(M+N) and a space complexity of O(M+N). Compared with related technologies, it can effectively reduce the time complexity and space complexity of the algorithm, thereby improving operation efficiency, improving text comparison efficiency, saving computing resources and time overhead, and improving work efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data analysis, and in particular to a text comparison method, device, electronic device and computer-readable storage medium. Background Art

[0002] When calculating the similarity of large-scale texts, the traditional solution is to use similarity comparison algorithms such as LD (edit distance) algorithm and Needlema-Wunsch algorithm. However, both LD algorithm and Needlema-Wunsch algorithm have high algorithm complexity and low operating efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a text comparison method, device, electronic device and computer-readable storage medium to improve the efficiency of text comparison.

[0004] An embodiment of the present application provides a text comparison method, including: traversing a first text to obtain a fingerprint value of a first fingerprint of each character string in the first text; traversing a second text to obtain a fingerprint value of a second fingerprint of each character string in the second text; comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain a similarity between the second text and the first text; wherein the fingerprint value is a value that characterizes the content and structural characteristics of a character string.

[0005] In the above implementation process, by traversing the first text and the second text, the fingerprint values ​​of each string in the first text and the second text are obtained, and since each fingerprint value depends on a string, whether the fingerprint value is different can reflect whether the string corresponding to the fingerprint value is different. Accordingly, by comparing the fingerprint value of the first fingerprint and the fingerprint value of the second fingerprint, the similarity between the second text and the first text can be effectively determined. The scheme of the embodiment of the present application only needs to traverse the first text and the second text and compare them, and the time complexity is about O(M+N) (M and N are the lengths of the strings of the two texts, and O is the complexity symbol), and the space complexity is O(M+N). The LD algorithm, Needlema-Wunsch algorithm and other algorithms in the related art have a time complexity and a space complexity of O(M*N), so the scheme of the embodiment of the present application can effectively reduce the time complexity and space complexity of the algorithm compared with the related art, thereby improving the operation efficiency, improving the text comparison efficiency, saving computing resources and time overhead, and improving work efficiency.

[0006] Furthermore, the fingerprint value of the first fingerprint is a hash value of the character string in the first text; and the fingerprint value of the second fingerprint is a hash value of the character string in the second text.

[0007] In the above implementation process, by using the hash value as the fingerprint value of the string, the implementation is simple and reliable, which is conducive to the promotion of the solution of the embodiment of the present application in industrial applications.

[0008] Furthermore, the hash value of the string in the first text is: the hash value calculated corresponding to the last character in the string of the first text; the hash value of the string in the second text is: the hash value calculated corresponding to the last character in the string of the second text; wherein the hash value of each character in the string is a value calculated based on the hash value of the previous character and its own unique identification value.

[0009] In the above implementation process, since the hash value of a string is the calculated hash value corresponding to the last character in the string, and the hash value of each character in the string needs to be calculated based on the hash value of the previous character and its own unique identification value, this makes the features of each character in the string accumulate backwards, so that in actual calculations, the difference between two characters with extremely small differences will be magnified, making the difference in the calculated hash values ​​larger, thereby ensuring the effect of subsequent comparisons.

[0010] Furthermore, the hash value of each character in the string is: The calculated value of f(x); where x represents the x-th character in the string, seed is a preset constant, and str[x] is the unique identifier of the x-th character in the string.

[0011] In the above implementation process, through the design of seed, when calculating the hash value of each character, there can be a greater difference, so that the difference in hash values ​​between different strings is greater, ensuring the subsequent comparison effect.

[0012] Furthermore, the unique identification value is the ASCII code value of the character.

[0013] In the above implementation process, using the ASCII code value of the character as the unique identification value of the character can effectively ensure the uniqueness of the hash value calculation for each character, thereby ensuring the uniqueness of the hash value of the string.

[0014] Furthermore, each character string in the first text is a character string composed of characters corresponding to each line in the first text; each character string in the second text is a character string composed of characters corresponding to each line in the second text.

[0015] In the above implementation process, a character string is constructed in line units and then compared, so that line-based file similarity comparison can be achieved, which is conducive to use in large files such as log files, data files, and codes.

[0016] Furthermore, the fingerprint value of the first fingerprint is compared with the fingerprint value of the second fingerprint to obtain the similarity between the second text and the first text, including: for each second fingerprint in the second text, the fingerprint value of the second fingerprint is compared with the fingerprint values ​​of each first fingerprint in the first text in turn; the number and position of first fingerprints having the fingerprint value of the second fingerprint in the first text are counted; and the similarity between the second text and the first text is determined based on the number and position of first fingerprints corresponding to each second fingerprint obtained by counting.

[0017] Furthermore, based on the number and positions of the first fingerprints obtained by statistics, the similarity between the second text and the first text is determined, including: constructing a similarity matrix based on the number and positions of the first fingerprints obtained by statistics; and calculating the similarity matrix using a preset similarity algorithm to obtain the similarity between the second text and the first text.

[0018] In this way, a sparse similarity matrix can be quickly generated, so that the time cost of calculating the similarity can be greatly reduced through the sparse similarity matrix.

[0019] An embodiment of the present application also provides a text comparison device, including: an acquisition module, used to traverse a first text, and obtain fingerprint values ​​of first fingerprints of each string in the first text; and used to traverse a second text, and obtain fingerprint values ​​of second fingerprints of each string in the second text; a comparison module, used to compare the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint, and obtain the similarity between the second text and the first text; wherein the fingerprint value is a value that characterizes the content and structural characteristics of the string.

[0020] An embodiment of the present application also provides an electronic device, including a processor, a memory and a communication bus; the communication bus is used to realize connection and communication between the processor and the memory; the processor is used to execute one or more programs stored in the memory to implement any of the above-mentioned text comparison methods.

[0021] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the above-mentioned text comparison methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A basic flow chart of a text comparison method provided in an embodiment of the present application;

[0024] Figure 2 A specific text comparison process diagram provided for an embodiment of the present application;

[0025] Figure 3 A structural schematic diagram of a text comparison device provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0028] Embodiment 1:

[0029] In practical applications, the complexity of an algorithm is usually evaluated using time complexity and space complexity. Time complexity is used to evaluate the time required to execute a program, which can estimate the program's use of the processor; space complexity is used to evaluate the storage space required to execute a program, which can estimate the program's use of computer memory.

[0030] In the field of text similarity comparison, the inventors have noticed that when calculating the similarity of large-scale texts, the traditional solution is to use similarity comparison algorithms such as the LD algorithm and the Needlema-Wunsch algorithm to achieve it.

[0031] Among them, the LD algorithm calculates similarity based on the edit distance, and transforms string A into another string B by inserting, deleting, and replacing characters, and indirectly quantifies the degree of difference between the two strings by recording the minimum number of operations. When using the LD algorithm, because the LD matrix is ​​used to calculate the matching path and obtain the specific matching string, the time complexity and space complexity of the LD algorithm are O(M*N), where M and N are the lengths of the two text strings. When the magnitude of the two matched texts is small, its performance is acceptable, but if the magnitude is too large, the algorithm's consumption of computing power and memory will increase exponentially. For example: if both strings have 1024 characters, the size of the LD matrix is ​​1024*1024*4B=400MB. It can be clearly seen that this level of storage space usage is not suitable for application scenarios with large-scale text files.

[0032] The Needlema-Wunsch algorithm is an LCS (text comparison) algorithm based on the longest common subsequence. Similar to the LD algorithm, the Needlema-Wunsch algorithm also uses the idea of ​​dynamic programming. In the Needlema-Wunsch algorithm, a weight is also set to distinguish the priorities of the three operations (insertion, deletion, and modification). Like the LD algorithm, the time complexity and space complexity of the Needlema-Wunsch algorithm are also O(M*N), and it is also not suitable for application scenarios with large-scale text files.

[0033] Then, in order to ensure the operating efficiency when calculating the similarity of large-scale texts and improve the efficiency of text comparison, a text comparison method is provided in the embodiment of the present application. Figure 1 As shown, Figure 1 The flowchart of the text comparison method provided in the embodiment of the present application includes:

[0034] S101: traverse a first text to obtain a fingerprint value of a first fingerprint of each character string in the first text.

[0035] S102: traverse the second text to obtain a fingerprint value of a second fingerprint of each character string in the second text.

[0036] It should be noted that, in the embodiment of the present application, the first text and the second text are two texts that need to be compared for similarity. When multiple texts need to be compared for similarity, any two texts can be compared for similarity using the method of the present application.

[0037] In the embodiment of the present application, the first text and the second text may be log files, data files, codes, configuration files, etc., which are not limited in the embodiment of the present application.

[0038] It should also be noted that in the embodiment of the present application, there is no time sequence restriction between step S101 and step S102. That is, between step S101 and step S102, step S101 may be executed first, or step S102 may be executed first, or step S101 and step S102 may be executed simultaneously, which is not limited in the embodiment of the present application.

[0039] It should also be noted that in the embodiments of the present application, the fingerprint value can characterize the content and structural characteristics of the string. That is, the difference in the number of characters in the string, the difference in the character content, and the difference in the arrangement and combination of characters will affect the fingerprint value corresponding to the string. Each fingerprint value depends on a string, so whether the fingerprint value is different can reflect whether the string corresponding to the fingerprint value is different.

[0040] Exemplarily, in a feasible implementation of the embodiment of the present application, the hash value of the string in the first text can be used as the fingerprint value of the first fingerprint, and the hash value of the string in the second text can be used as the fingerprint value of the second fingerprint.

[0041] In an embodiment of the present application, the hash value of a string can be calculated using algorithms such as MD5 (Message Digest Algorithm, message digest algorithm version 5), SHA (Secure Hash Algorithm, secure hash algorithm), DES (Data Encryption Standard, data encryption standard) algorithm, AES (Advanced Encryption Standard, advanced encryption standard) algorithm, etc.

[0042] In addition, in an optional implementation of an embodiment of the present application, the hash value corresponding to each character in the string can also be calculated, and when calculating, the hash value corresponding to the current character can be determined by the hash value of the previous character and the unique identification value of the current character itself (it should be noted that for the first character in the string, since there is no hash value of the previous character, its own hash value can be determined only based on its own unique identification value).

[0043] Since the hash value of the last character is obtained based on the hash values ​​of the previous characters, the hash value of the last character will be affected by all the characters in the string. Any change in any of the previous characters will cause the hash value of the last character to change. Therefore, in a feasible example, the hash value of the last character in the string can be used as the hash value of the string, so that the string can be uniquely represented.

[0044] For example, assuming that a string consists of three characters A, B, and C in sequence, you can first calculate the hash value a corresponding to character A based on the unique identification value of character A, and then calculate the hash value b corresponding to character B based on the hash value a and the unique identification value of character B (this is actually equivalent to calculating the hash value of string AB), and then calculate the hash value c corresponding to character C based on the hash value b and the unique identification value of character C (this is actually equivalent to calculating the hash value of string ABC), and finally, the hash value c can be used as the hash value of the string.

[0045] It should be noted that in the embodiments of the present application, the unique identification value of a character refers to a value that can uniquely identify a character. For example, it can be the ASCII code value of the character. For another example, it can be a number that uniquely corresponds to each character pre-written by an engineer.

[0046] In another feasible example of the embodiment of the present application, the hash value of the string can also be further determined based on the hash value of each character in the string. For example, the hash value of each character in the string can be hashed again to obtain the hash value of the string.

[0047] It should be understood that in the above two examples, when calculating the hash value of each character in the string of characters, an optional calculation method is:

[0048] According to the formula Calculate the hash value of each character.

[0049] Here, x represents the xth character in the string, seed is a preset random number, and str[x] is the unique identification value of the xth character in the string.

[0050] It should be understood that in the above calculation method, the seed can be a random number pre-selected by an engineer. In addition, the random number (i.e., the seed) can be an odd number greater than 1, thereby reducing the risk of different strings having the same hash value and ensuring the reliability of the solution.

[0051] In this way, the fingerprint value of the same string is also the same, otherwise an "avalanche effect" will occur and even tiny text differences will be magnified.

[0052] It should be noted that in practical applications, both the first text and the second text are sets of a large number of characters, and the character strings in the first text and the second text can be obtained by pre-setting a division rule.

[0053] Exemplarily, each line of characters in the first text and the second text may be determined as a character string in units of lines, and then step S101 and step S102 are performed.

[0054] Of course, it can also be divided in other ways, such as dividing by paragraphs, or dividing by n (n is a pre-set positive integer greater than 1) words. The division method is not limited in the embodiments of the present application.

[0055] It should also be noted that in actual applications, when calculating the hash value of a string, the calculated hash value may overflow (that is, the number of bits required for the hash value exceeds the maximum number of bits of the computer), which may cause the calculated hash value to be negative. To solve this problem, the calculated hash value can be ANDed with the value represented by the maximum number of bits of the computer to ensure that the obtained hash value is positive, which is convenient for subsequent use.

[0056] For example, taking a 32-bit computer system as an example, if the calculated hash value overflows, it can be ANDed with 0x7fffffff to obtain a positive number.

[0057] In the above process, the time complexity of traversing to obtain the fingerprint value of each string in the first text and the second text is O(M+N) (where M is the total length of the strings of the first text and M is the total length of the strings of the second text), which will not cause a bottleneck in performance.

[0058] S103: Compare the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain the similarity between the second text and the first text.

[0059] In an embodiment of the present application, for each second fingerprint in the second text, the fingerprint value of the second fingerprint can be compared with the fingerprint values ​​of each first fingerprint in the first text in turn, and then the number and position of the first fingerprints with the fingerprint value of the second fingerprint in the first text are counted, and the similarity between the second text and the first text is determined based on the number and position of the first fingerprints corresponding to each second fingerprint obtained by counting.

[0060] For example, a similarity matrix can be constructed based on the number and position of the first fingerprints having the fingerprint value of the second fingerprint in the first text obtained by statistics, and then a preset similarity algorithm (such as the Dijkstra algorithm, the dynamic programming algorithm, etc.) is used to calculate the similarity matrix to obtain the similarity between the second text and the first text.

[0061] For example, assume that the texts to be compared are:

[0062] Text 1:

[0063] AAAAAAAAAAAAAAAAAAA

[0064] BBBBBBBBBBBBBBBBBBBBBB

[0065] CCCCCCCCCCCCCCCCCCCCCCCC

[0066] DDDDDDDDDDDDDDDDDDDD

[0067] EEEEEEEEEEEEEEEEEEEEEEEEEE

[0068] Text 2:

[0069] AAAAAAAAAAAAAAAAAAA

[0070] bBBBBBBBBBBBBBBBBBBBBBB

[0071] CCCCCCCCCCCCCCCCCCCCCCCC

[0072] DDDDDDDDDDDDDDDDDD

[0073] EEEEEEEEEEEEEEEEEEEEEEEEEE

[0074] Assume that the calculated fingerprint value array of text 1 is {1, 2, 3, 4, 5}, and the calculated fingerprint value array of text 2 is {1, 6, 3, 7, 5}.

[0075] For the fingerprint values ​​1, 6, 3, 7, and 5 of text 2, we query whether they exist in the fingerprint value array of file 1 in turn. We can get: fingerprint values ​​1, 3, and 5 exist and the quantity is 1, and the positions are 1, 3, and 5 respectively. Fingerprint values ​​6 and 7 do not exist and the quantity is 0, so the similarity matrix is:

[0076] {1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0

[0080] 0 0 0 0 1}

[0081] Then, the Dijkstra algorithm is used to calculate the maximum value path of the matrix, which is the diagonal "10101" in the matrix, so that the similarity between text 2 and text 1 can be calculated to be 3. Among them, the Dijkstra algorithm is a general algorithm for calculating the optimal path, which will not be described in detail here.

[0082] In the embodiment of the present application, in order to facilitate comparison, the fingerprint value of the first fingerprint can be recorded in a fingerprint table, and the fingerprint value of each second fingerprint can be stored in an array, and then the data in the array can be traversed, and the fingerprint value taken out from the array can be used to search for its existence in the fingerprint table, and the number C1 and position of the first fingerprints with the same fingerprint value can be counted. Repeat this process until the array is traversed, that is, the number and position of the first fingerprints corresponding to all the second fingerprints are obtained, and the similarity between the second text and the first text is generated accordingly.

[0083] It should be understood that the above method of obtaining the similarity between the second text and the first text is only a feasible implementation method illustrated in the embodiment of the present application. In addition, other methods can also be used in the embodiment of the present application. The embodiment of the present application does not limit the specific implementation method of step S103.

[0084] The text comparison method provided in the embodiment of the present application obtains the fingerprint value of each string in the first text and the second text by traversing the first text and the second text. Since each fingerprint value depends on a string, whether the fingerprint values ​​are different can reflect whether the strings corresponding to the fingerprint values ​​are different. Accordingly, by comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint, the similarity between the second text and the first text can be effectively determined.

[0085] The solution of the embodiment of the present application only needs to traverse the first text and the second text and compare them, and the time complexity is about O(M+N) (M and N are the lengths of the string arrays of the two texts, and O is the complexity symbol), and the space complexity is O(M+N). The LD algorithm, Needlema-Wunsch algorithm and other algorithms in the related art have a time complexity and space complexity of O(M*N). Therefore, compared with the related art, the solution of the embodiment of the present application can effectively reduce the time complexity and space complexity of the algorithm, thereby improving the operation efficiency, improving the text comparison efficiency, saving computing resources and time overhead, and improving work efficiency.

[0086] Embodiment 2:

[0087] Based on the first embodiment, this embodiment takes a specific implementation process of performing similarity comparison on two texts as an example to further illustrate this application.

[0088] See also Figure 2 As shown, including:

[0089] S201: Read the character string in file 1 line by line.

[0090] S202: Verify the read character string to determine whether the file 1 ends.

[0091] Exemplarily, if the read string is empty, it can be considered that the file 1 has been read and the file 1 has ended. At this time, the process can go to step S204. If the read string is not empty, it can be considered that the file 1 has not ended. At this time, the process can go to step S203.

[0092] S203: The read character string is processed according to the formula Calculate the hash value, add it to the hash table (referred to as HashList1), create an index, and then go to step S201.

[0093] Here, x represents the x-th character in the string, seed is a preset odd number greater than 1, and str[x] is the ASCII code value of the x-th character in the string.

[0094] S204: Read the character string in file 2 line by line.

[0095] S205: Verify the read character string to determine whether the file 2 ends.

[0096] If the file 2 ends, the process may go to step S207. If the file 2 does not end, the process may go to step S206.

[0097] The verification method is the same as step S202 and will not be described in detail here.

[0098] S206: The read string is processed according to the formula Calculate the hash value and store it in an array (referred to as HashArr2). Then go to step S204.

[0099] S207: Read the current data from the array HashArr2.

[0100] S208: Determine whether the currently read data is empty.

[0101] If it is empty, it means that the data in HashArr2 has been read, and go to step S210; if it is not empty, go to step S209.

[0102] S209: Use the hash value taken from HashArr2 to search whether it exists in HashList1, and count the number of times C1 and the position. Go to step S207.

[0103] S210: Summarize and statistically analyze the C1 values ​​and positions corresponding to the hash values ​​in each HashArr2, and generate a calculation similarity result.

[0104] Through the above scheme, the time complexity and space complexity of the algorithm can be controlled to the level of O(M+N), thereby improving the operation efficiency, improving the text comparison efficiency, saving computing resources and time overhead, and improving work efficiency. See the following table, which is the comparison test results provided by this application with the Needleman-Wunsch algorithm:

[0105]

[0106]

[0107] As can be seen from the above table, the solution of the embodiment of the present application reduces the average time consumption by about 90%. For large-scale text files, the improvement is even more obvious, reducing the time consumption by nearly 99%, which has a significant efficiency improvement advantage.

[0108] Embodiment three:

[0109] Based on the same inventive concept, the present application also provides a text comparison device 300. Figure 3 As shown, Figure 3 Shows the use of Figure 1 It should be understood that the specific functions of the device 300 can be found in the description above, and the detailed description is appropriately omitted here to avoid repetition. The device 300 includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system of the device 300. Specifically:

[0110] See also Figure 3 As shown, the device 300 includes: an acquisition module 301 and a comparison module 302. Among them:

[0111] The acquisition module 301 is used to traverse the first text to obtain the fingerprint value of the first fingerprint of each character string in the first text; and to traverse the second text to obtain the fingerprint value of the second fingerprint of each character string in the second text;

[0112] The comparison module 302 is used to compare the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain the similarity between the second text and the first text;

[0113] The fingerprint value is a value that represents the content and structural characteristics of the character string.

[0114] In a feasible implementation manner of the embodiment of the present application, the fingerprint value of the first fingerprint is a hash value of the character string in the first text; the fingerprint value of the second fingerprint is a hash value of the character string in the second text.

[0115] In an optional example scheme of this feasible implementation mode, the hash value of the string in the first text is: the hash value calculated corresponding to the last character in the string of the first text; the hash value of the string in the second text is: the hash value calculated corresponding to the last character in the string of the second text; wherein the hash value of each character in the string is: a value calculated based on the hash value of the previous character and its own unique identification value.

[0116] In this optional example, the hash value of each character in the string is: The calculated value of f(x); where x represents the x-th character in the string, seed is a preset random number, and str[x] is the unique identifier of the x-th character in the string.

[0117] In the embodiment of the present application, the unique identification value is the ASCII code value of the character.

[0118] In an embodiment of the present application, each character string in the first text is a character string composed of characters corresponding to each line in the first text; each character string in the second text is a character string composed of characters corresponding to each line in the second text.

[0119] In an embodiment of the present application, the comparison module 302 is specifically used to compare the fingerprint value of each second fingerprint in the second text with the fingerprint values ​​of each first fingerprint in the first text in turn, count the number and position of first fingerprints with the fingerprint value of the second fingerprint in the first text, and determine the similarity between the second text and the first text based on the number and position of first fingerprints corresponding to each second fingerprint obtained by counting.

[0120] In the embodiment of the present application, the comparison module 302 is specifically used to construct a similarity matrix according to the statistically obtained number and position of the first fingerprints; and calculate the similarity matrix using a preset similarity algorithm to obtain the similarity between the second text and the first text.

[0121] It should be understood that, for the sake of brevity, some of the contents described in the first embodiment will not be repeated in this embodiment.

[0122] Embodiment 4:

[0123] This embodiment provides an electronic device, see Figure 4 As shown, it includes a processor 401, a memory 402 and a communication bus 403. Among them:

[0124] The communication bus 403 is used to realize the connection and communication between the processor 401 and the memory 402 .

[0125] The processor 401 is used to execute one or more first programs stored in the memory 402 to implement the text comparison method described in the above-mentioned embodiment 1 and / or embodiment 2.

[0126] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 For example, the electronic device may also have input and output components such as a mouse, a keyboard, etc., may also have display components such as a display screen, and may also have external communication components such as an antenna, a USB bus, etc.

[0127] It should be understood that the electronic device described in the embodiments of the present application may be a device such as a computer, a server, etc. that has data comparison and processing capabilities.

[0128] This embodiment also provides a computer-readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a U disk, an SD (Secure Digital Memory Card) card, an MMC (Multimedia Card) card, etc., in which one or more programs for implementing the above steps are stored, and the one or more programs can be executed by one or more processors to implement the steps of the text comparison method performed by the service distribution device in the above embodiment 1 and / or embodiment 2, or to implement the steps of the text comparison method performed by the processing node in the above embodiment 1 and / or embodiment 2. No further details will be given here.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0130] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0132] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0133] As used herein, a plurality refers to two or more than two.

[0134] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A text comparison method, characterized in that: include: Traversing the first text, obtaining a fingerprint value of a first fingerprint of each character string in the first text; Traversing the second text, obtaining a fingerprint value of a second fingerprint of each character string in the second text; Comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain a similarity between the second text and the first text; The fingerprint value is a value that represents the content and structural characteristics of the character string; The fingerprint value of the first fingerprint is a hash value of the string in the first text; the fingerprint value of the second fingerprint is a hash value of the string in the second text; The hash value of the string in the first text is: the calculated hash value corresponding to the last character in the string of the first text; The hash value of the string in the second text is: the calculated hash value corresponding to the last character in the string of the second text; The hash value of each character in the string is a value calculated based on the hash value of the previous character and its own unique identification value.

2. The text comparison method according to claim 1, characterized in that: The hash value of each character in the string is: According to the formula The calculated value of f(x); Among them, x represents the x-th character in the string, deed is a preset constant, and str[x] is the unique identification value of the x-th character in the string.

3. The text comparison method according to claim 2, characterized in that: The unique identification value is the ASCII code value of the character.

4. The text comparison method according to any one of claims 1 to 3, characterized in that: Each character string in the first text is a character string composed of characters corresponding to each line in the first text; Each character string in the second text is a character string composed of characters corresponding to each line in the second text.

5. The text comparison method according to any one of claims 1 to 3, characterized in that: Comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain the similarity between the second text and the first text includes: For each second fingerprint in the second text, comparing the fingerprint value of the second fingerprint with the fingerprint values ​​of each first fingerprint in the first text in sequence; Counting the number and positions of first fingerprints having the fingerprint value of the second fingerprint in the first text; The similarity between the second text and the first text is determined based on the number and positions of the first fingerprints obtained by statistics.

6. The text comparison method according to claim 5, characterized in that: Determining the similarity between the second text and the first text according to the number and position of the first fingerprints obtained by statistics includes: constructing a similarity matrix according to the number and positions of the first fingerprints obtained by statistics; The similarity matrix is ​​calculated using a preset similarity algorithm to obtain the similarity between the second text and the first text.

7. A text comparison device, characterized in that: include: An acquisition module, configured to traverse the first text and acquire a fingerprint value of a first fingerprint of each character string in the first text; and for traversing the second text to obtain the fingerprint value of the second fingerprint of each character string in the second text; A comparison module, used for comparing the fingerprint value of the first fingerprint with the fingerprint value of the second fingerprint to obtain the similarity between the second text and the first text; The fingerprint value is a value that represents the content and structural characteristics of the character string; The fingerprint value of the first fingerprint is a hash value of the string in the first text; the fingerprint value of the second fingerprint is a hash value of the string in the second text; The hash value of the string in the first text is: the calculated hash value corresponding to the last character in the string of the first text; The hash value of the string in the second text is: the calculated hash value corresponding to the last character in the string of the second text; The hash value of each character in the string is a value calculated based on the hash value of the previous character and its own unique identification value.

8. An electronic device, characterized in that: include: Processor, memory and communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute the program stored in the memory to implement the text comparison method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the text comparison method according to any one of claims 1 to 6.

Images