Text comparison method, system and computer program product thereof

Abstract

A text comparison method is suitable for comparing a document to be inspected with an existing document. The comparison method includes: performing irreversible conversion on the existing document to obtain a first intermediate file, the first intermediate file includes a plurality of characters, and the number of different characters is a predetermined value; receiving a second intermediate file, the second intermediate file is a file of the document to be inspected after irreversible conversion; and, according to a predetermined length, using a high-repetition character comparison method, comparing the second intermediate file with the first intermediate file to output a comparison result.

Description

Technical Field

[0001] The present invention relates to a text comparison method, system and computer program product thereof, and in particular to a text comparison method, system and computer program product thereof capable of maintaining privacy. Background Art

[0002] Currently, plagiarism detection services require high fees, and the database maintenance costs of plagiarism detection service providers are quite high, and the accuracy of plagiarism judgment is low. In addition, if users want to use plagiarism judgment services, they need to send their files to plagiarism judgment service providers. For confidential data, users will inevitably have concerns about confidential data leakage. Summary of the invention

[0003] In view of this, some embodiments of the present invention provide a text comparison method, system and computer program product capable of maintaining privacy.

[0004] According to some embodiments, a text comparison method is suitable for comparing a document to be inspected with an existing document. The comparison method includes: performing an irreversible conversion on the existing document to obtain a first intermediate file, the first intermediate file includes a plurality of characters, and the number of different characters is a predetermined value; receiving a second intermediate file, the second intermediate file is a file of the document to be inspected after irreversible conversion; and, according to a predetermined length, using a high-repetition character comparison method, comparing the second intermediate file with the first intermediate file to output a comparison result.

[0005] According to some embodiments, the irreversible conversion includes: each character of the corresponding converted file is one of a plurality of different characters, and a plurality of corresponding characters are integrated to output a first intermediate file, wherein the corresponding conversion includes: obtaining a standard encoding corresponding to the character; obtaining a remainder of the standard encoding divided by a predetermined value; and obtaining a corresponding character in a lookup table based on the remainder.

[0006] According to some embodiments, the lookup table includes index values ​​and a plurality of different characters corresponding to respective index values, and the index value is an integer from zero to a value less than a predetermined value.

[0007] According to some embodiments, a computer program product includes a set of instructions, and when a computer loads and executes this set of instructions, it can complete the text comparison method according to any embodiment of the present invention.

[0008] According to some embodiments, a file comparison system is suitable for comparing a file to be inspected with an existing file. The file comparison system includes a communication module and an operation unit. The operation unit is used to perform an irreversible conversion on the existing file to obtain a first intermediate file, wherein the first intermediate file includes a plurality of characters, and the number of different characters is a predetermined value. And the operation unit compares the second intermediate file with the first intermediate file according to a predetermined length using a high-repetition character comparison method to output a comparison result. The communication module is electrically connected to the operation unit. The operation unit receives the second intermediate file through the communication module, and the second intermediate file is a file obtained by irreversibly converting the file to be inspected.

[0009] The following is a detailed description of the specific embodiments in conjunction with the attached drawings, so that the purpose, technical content, characteristics and effects achieved by the present invention can be more easily understood. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram illustrating steps of a text comparison method according to some embodiments of the present invention is shown.

[0011] Figure 2 A schematic diagram of the steps of text comparison according to some embodiments of the present invention is shown.

[0012] Figure 3 A functional block diagram of a text matching system according to some embodiments of the present invention is shown.

[0013] Figure 4 A schematic diagram illustrating steps of a text comparison method according to some embodiments of the present invention is shown.

[0014] Figure 5 A schematic diagram illustrating steps of a text comparison method according to some embodiments of the present invention is shown.

[0015] Figure 6 A schematic diagram illustrating steps of a text comparison method according to some embodiments of the present invention is shown.

[0016] Figure 7 A circuit block diagram of a text matching system according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0017] The following will describe various embodiments of the present invention in detail, with accompanying drawings as examples. In the description of the specification, many specific details are provided in order to give readers a more complete understanding of the present invention; however, the present invention may still be implemented under the premise of omitting some or all of the specific details. The same or similar components in the drawings will be represented by the same or similar symbols. It is particularly noted that the drawings are for illustration only and do not represent the actual size or number of components. Some details may not be fully drawn to simplify the drawings.

[0018] Please refer to Figure 1 , Figure 1 A schematic diagram of the steps of the text comparison method of some embodiments of the present invention is shown. The text comparison method is suitable for comparing the document to be checked with the existing document to determine whether part of the content in the document to be checked is the same as part of the content in the existing document. In some embodiments, the text comparison method is applied to plagiarism detection. For example, the document to be checked is a newly completed paper or article, and the existing document is a published document. The published document can be one or more articles. The text comparison method compares the document to be checked with the existing document to determine whether the document to be checked is plagiarized or plagiarized from the existing document. After the comparison, the text comparison method outputs the comparison result. The content of the comparison result will be described in detail later.

[0019] Please refer to Figure 1 According to some embodiments, the text comparison method includes:

[0020] Step S1: performing irreversible conversion on the existing file to obtain a first intermediate file, wherein the first intermediate file includes a plurality of characters, and the number of the different characters is a predetermined value;

[0021] Step S2: receiving a second intermediate file, wherein the second intermediate file is a file obtained by irreversibly converting the file to be inspected; and

[0022] Step S3: According to a predetermined length, the second intermediate file is compared with the first intermediate file by a high-repetition character comparison method to output a comparison result.

[0023] The irreversible conversion in step S1 refers to converting each "word" in the original file and outputting a converted file, and the converted file cannot be restored to the original file. For example, the original file is a file, and the file includes a plurality of "characters". After the irreversible conversion, the first intermediate file is output, and the first intermediate file includes a plurality of "characters". The number of different characters is a predetermined value q. For example, the character type can be but not limited to a combination of a single or multiple categories such as letters, punctuation marks and special symbols, but the number of different character types used is a predetermined value, for example: A, +, and @ are taken as 3 different character types with a limited number. For another example: the irreversible conversion system converts each character in the original file into a character according to a many-to-one function relationship (surjective function) to output an intermediate file, wherein the predetermined value q is but not limited to the number of character types such as 4, 6, 8, 12, 16, 20, etc. In some embodiments, the predetermined value q is 4, and the different characters are A, T, C, and G respectively. Each character of the intermediate file is selected from a group of different characters, for example, the content of the intermediate file is ATTAAACCGATTAGGACCC (having the four different characters), or AGGGTTAAAGGTTT (having only the characters A, T, G, but no character C). Therefore, the characters of the intermediate file have a highly repetitive characteristic, that is, when the total number of characters in the file is 3,000 words, the intermediate file after irreversible conversion only includes the four different characters, so its characters are highly repeated in the intermediate file. Therefore, the character repetitiveness of the intermediate file with a total number of 10,000 characters is higher than the character repetitiveness of the intermediate component with a total number of 3,000 characters. In some embodiments, the above-mentioned characters are single words (words) or letters (alphabets), and the data volume and standard encoding of the two can be adjusted as needed.

[0024] In some embodiments, the aforementioned irreversible conversion is a standard coding remainder method, which means that the standard coding of each character of the original file before conversion is divided by the aforementioned predetermined value q, and the remainder is taken. Then the corresponding character is given according to the remainder. In some embodiments, the aforementioned standard coding is but not limited to ASCII code (American Standard Code for Information Interchange) or Chinese telegraph code. The standard coding (ASCII code) of character A is 65 (decimal, the same below), the standard coding (ASCII code) of character E is 69, the standard coding (ASCII code) of character T is 84, and the standard coding (Chinese telegraph code) of character Yang is 26954. The predetermined value q is explained by taking 4 as an example, and the remainder may be 0, 1, 2, 3. The characters corresponding to each remainder are assumed to correspond to A, T, C, and G respectively. Therefore, the character A of the original English file, after irreversible conversion, obtains the character T. The standard coding of the character "Yang" in the original file is divided by the aforementioned predetermined value, and the remainder is 2, which corresponds to the character C. The rest are similar and will not be repeated. In some embodiments, the standard code remainder method is to divide the standard code of a word or a single word by the predetermined value, and obtain the remainder to obtain the corresponding character. For example, the single word EAT in the original English document, after irreversible conversion, the sum of the standard codes of the three letters E, A, and T contained in the single word "EAT" is 218. Divide 218 by the predetermined value 4, and the remainder is 2, which corresponds to the character C.

[0025] In some embodiments, the characters used in the document to be checked and the existing document are non-English characters, such as but not limited to Chinese, Japanese or Korean. Therefore, when performing the irreversible conversion of the standard coding remainder method, the standard coding is the corresponding standard coding of the Chinese, Japanese or Korean characters in the computer.

[0026] Specifically, the irreversible conversion of the aforementioned standard coding remainder method includes: each character of the corresponding converted file is one of the different characters, and the corresponding characters are integrated to output the first intermediate file. Figure 2 In one embodiment, the corresponding conversion includes:

[0027] Step S10: Obtain the standard code corresponding to the character. If the character contains multiple letters, the sum of the standard codes of the letters can be used as the standard code of the character, but is not limited to;

[0028] Step S12: Obtain the remainder of dividing the standard code by a predetermined value q; and

[0029] Step S14: Obtain a corresponding character from a lookup table according to the remainder.

[0030] The aforementioned lookup table can be, but is not limited to, a remainder corresponding character lookup table, such as: index values ​​0, 1, 2, 3 correspond to A, T, C, G respectively. In some embodiments, the lookup table can be defined by the user or its correspondence can be determined by a random number, which will further increase its irreversible effect. However, the lookup table in the same text matching method must be the same. In some embodiments, the lookup table includes index values ​​and the different characters corresponding to each index value, and the index value is an integer from zero to less than a predetermined value. Other implementations of irreversible conversion will be described later.

[0031] The receiving in step S2 can be realized by external transmission methods such as wireless network or wired network, or by reading the stored data file internally. Figure 3 , taking the wireless network transmission method as an example, the local terminal device 1 performs an irreversible conversion on the file to be inspected to obtain a second intermediate file, and the local terminal device transmits the second intermediate file to the text comparison system 2 on the server side through the Internet. Therefore, hackers cannot obtain the original file to be inspected through the Internet. This can be called a private receiving mode. In some embodiments, the text comparison system 2 on the server side receives the original file to be inspected transmitted by the local terminal device 1. The text comparison system 2 performs an irreversible conversion on the file to be inspected to output a second intermediate file. In the process of the terminal device 1 transmitting through the Internet, hackers may steal the original file to be inspected through the Internet. This can be called a public receiving mode. In addition, taking the wired transmission method as an example, the user can use a portable storage device such as a flash drive to directly transmit the original file to be inspected to the text comparison system 2, but it is not limited to this.

[0032] It is worth noting that the above-mentioned private receiving mode or public receiving mode refers to whether the original file to be checked is exposed to the risk of being stolen by hackers, rather than the second intermediate file. It should be noted that as far as the second intermediate file is concerned, even if it is illegally stolen by hackers, the original content of the file to be checked cannot be restored. This is due to the effect of the aforementioned irreversible conversion. In other words, this embodiment achieves the effect of privacy comparison through the second intermediate file.

[0033] The predetermined length in step S3 may be, for example but not limited to, a string consisting of k characters as a length unit. In at least one embodiment, the high-repetition character matching method outputs a matching result, such as but not limited to a JSON file, by sequentially matching strings of a predetermined length based on the second intermediate file and the first intermediate file. For example, the matching result indicates that there is no identical string or at least one hit range, so as to determine whether there is plagiarism. Other implementations of the high-repetition character matching method will be described later.

[0034] In some embodiments, if the document to be checked has a part of the content plagiarized, the comparison result includes the first character hit interval of the first intermediate file, thereby showing the paragraph where the content of the document is plagiarized; similarly, the comparison result may also include the second character hit interval of the second intermediate file, thereby showing the paragraph where the document to be checked involves plagiarism. That is, the comparison result is at least one of the first character hit interval of the first intermediate file and the second character hit interval of the second intermediate file. In some embodiments, the text comparison method may default to when the number of consecutive identical characters in the first intermediate file and the second intermediate file reaches the plagiarism threshold, such as but not limited to 8, that is, it is determined to be plagiarism. For example, the intermediate file contains q=8 types of different characters. The expected value of 8 consecutive identical characters is (1 / 88)=0.00000596%, which is very small, so it can be reasonably determined to be plagiarized content. The plagiarism threshold can be set, for example, based on the number of characters in the second intermediate file. Regarding the design and combination of the plagiarism threshold and the predetermined value q of different characters, those with ordinary knowledge can modify and change it by themselves, and should not be limited to this embodiment.

[0035] Other embodiments of the text comparison method are illustrated below.

[0036] In some embodiments, the text comparison method performs an irreversible conversion on an original file such as an existing file or a file to be inspected, and then outputs a position index file (Metafile). The position index file is used to correspond the "location" of each character in the converted intermediate file to the "location" of the corresponding original text in the original file; for example, the existing file is also subjected to irreversible conversion to obtain a first position index file, which is used to correspond each character position of the first intermediate file to each character position of the existing file, and the file to be inspected is also subjected to irreversible conversion to obtain a second position index file, which is used to correspond each character position of the second intermediate file to each character position of the file to be inspected. Thus, in some embodiments, the comparison result includes the position correspondence between the first character hit interval of the first intermediate file and the character hit interval of the existing file, and the position correspondence between the second character hit interval of the second intermediate file and the character hit interval of the file to be inspected.

[0037] In some embodiments, original files such as documents and documents to be inspected include multiple characters. After the text comparison method performs irreversible conversion, each character is encoded into a corresponding character to output an intermediate file, wherein each character is selected from a group consisting of multiple characters of a predetermined value q, and each character is assigned a corresponding index value, so as to establish a lookup table corresponding to the index value to the character, as shown in the following table, that is, the lookup table includes index values ​​and different characters corresponding to the respective index values, wherein the index value is an integer from zero to less than the predetermined value q.

[0038] Table 1

[0039] INTEGER_TO_CHAR_MAPPING=\

[0040] {0:'A',1:'R',2:'N',3:'D',4:'C',

[0041] 5:'Q',6:'E',7:'G',8:'H',9:'I',

[0042] 10:'L',11:'K',12:'M',13:'F',14:'P',

[0043] 15:'S',16:'T',17:'W',18:'Y',19:'V'}

[0044] When q=4, {0:'A', 1:'R', 2:'N', 3:'D'} or {0:'A', 1:'T', 2:'C', 3:'G'} are used.

[0045] When q=8, {0:'A', 1:'R', 2:'N', 3:'D', 4:'C', 5:'Q', 6:'E', 7:'G'} are used.

[0046] When q=16, {0:'A', 1:'R', 2:'N', 3:'D', 4:'C', 5:'Q', 6:'E', 7:'G', 8:'H', 9:'I', 10:'L', 11:'K', 12:'M', 13:'F', 14:'P', 15:'S'} are used.

[0047] For example: the number of different characters q is 8, and the different characters are A, R, N, D, C, Q, E, G respectively. Each character of the intermediate file is selected from the group consisting of 8 different characters. For example: the content of the first intermediate file is ARGQNCCGAEAGGADDD (with 8 different characters), or AGGGQQAAAGGNNN (only 4 characters such as A, N, Q, G are converted, and no other characters appear yet). Therefore, the intermediate file is a quantitative character file composed of highly repetitive characters, wherein the number of different characters q is a predetermined value, which can be but is not limited to: q=4, 6, 8, 12, 16 or 20; for example, when the number of different characters q is 4, the different characters are A, C, G, T respectively, and the character repetitiveness of the intermediate file composed thereof is higher. The following examples illustrate relevant embodiments of irreversible conversion.

[0048] In some embodiments, for Western languages, each letter corresponds to a standard encoding (Unicode), such as but not limited to: ASCII code. Thereby, the text comparison method can operate on a plurality of standard encodings corresponding to a plurality of letters contained in a character to generate a standard encoding corresponding to the character. For example, a character x contains n letters, xi represents the i-th letter in the character x, and ASCII (xi) represents the ASCII code of xi. As mentioned above, the intermediate file contains q different character types, for example, q = 8 means that the number of different characters is 8. Each character of the intermediate file is selected from a group of 8 different characters, and each character is assigned a corresponding index value from 0 to 7, so that a lookup table corresponding to the index value to the character can be established, for example {0: 'A', 1: 'R', 2: 'N', 3: 'D', 4: 'C', 5: 'Q', 6: 'E', 7: 'G'}, that is, the lookup table includes index values ​​and different characters corresponding to the respective index values, where the index value is an integer from zero to less than a predetermined value q.

[0049] PBS_char(x) represents the index value corresponding to the character x, and we specify that PBS_char(x) can be, for example but not limited to, a certain Char_to_number(x) function, and its possible calculation formula is formula (1):

[0050] PBS_char(x)=ROUND(i=1 to n Char_to_number(xi)*g(i))mod8......Formula (1),

[0051] Among them, Char_to_number(xi) can be, but is not limited to, ASCII(xi), BIG5(xi), UTF-8(xi), or Unicode(xi), etc. g(i) can be a constant value or a function of i, including but not limited to simple functions such as i*a + b and a*ib, where a is any positive real number and b is 0 or any positive real number; the ROUND function rounds a number to its nearest integer; the mod function takes the remainder after dividing the preceding number by the following number. The remainder obtained by dividing the positive integer before mod by 8 is the index value PBS_char(x) corresponding to the character x, and then the corresponding character is obtained by referring to the index value in the above lookup table. According to the above calculation results, PBS_char(x) is the remainder between 0 and 7 as the index value, respectively corresponding to characters such as A, R, N... G. For example, the letters e, a, t included in the English character "eat_" correspond to the standard codes 101, 97, 116 respectively. Substituting into Equation (1), we get PBS_char(x) = ROUND(i = 1 to 3 ASCII(xi)*1) mod 8 = 2. Through the above lookup table, the corresponding character obtained is "N_".

[0052] However, each character of oriental languages such as Chinese, Japanese, and Korean corresponds to a standard code respectively. Thus, the standard codes corresponding to each character can be directly and irreversibly converted to output an intermediate file, and its possible arithmetic formula is Equation (2):

[0053] PBS_char(y) = Char_to_number(y) mod 8 ………… Equation (2),

[0054] where Char_to_number(y) can be, but is not limited to, ASCII(y), BIG5(y), UTF-8(y), or Unicode(y), etc. The remainder obtained by dividing the standard code before mod by a predetermined value (q = 8) is referred to the aforementioned lookup table to obtain the character corresponding to the character y. According to the above calculation results, PBS_char(V) is the remainder between 0 and 7 as the index value, respectively corresponding to characters such as A, R, N... G, as described above. For example, the Chinese character "杨_" corresponds to the standard code 26954. Substituting into Equation (2), we get PBS_char(y) = ASCII(y) mod 8 = 2. Through the aforementioned lookup table, the corresponding character obtained is "N".

[0055] In some embodiments, oriental languages and western languages can share an arithmetic formula such as Equation (3) by using other types of one-to-many function relationships:

[0056] ROUND(i = 1 to n Char_to_number(xi)*g(i))............Equation (3)

[0057] After converting the Char_to_number(xi) of the character, such as the ASCII code, from decimal to binary value, take the first three digits as the representative value, and then refer to the aforementioned lookup table with the remainder obtained by dividing the representative value by the predetermined value to obtain the character corresponding to the character. Taking Chinese as an example, the decimal standard code corresponding to the character "Yang_" is 26954. After converting it to binary representation as 110100101001010, take the first three codes of the highest digit, that is, 110 as the representative value, and then through the remainder operation and the lookup table, it can be corresponded to the character "E" with an index value of 7; taking English as another example, the letters e, a, t included in the character "eat" are respectively corresponded to the standard codes 101, 97, 116. Substituting into Equation (3) gives the decimal operation result of 314, which is converted to binary representation as 100111010. Taking the first three codes of the highest digit, that is, 100 as the representative value, and then through the remainder operation and the lookup table, it can be corresponded to the character "C" with an index value of 5, but it is not limited to the above calculation formula.

[0058] In some embodiments, the Eastern language and the Western language can use other types of one-to-many function relationships for corresponding conversion, where the corresponding conversion includes: obtaining the number of digits according to the predetermined value; obtaining the remainder of the representative code according to the number of digits and the standard code; and obtaining the corresponding character in the lookup table according to the representative code. For example, after converting the standard code of the character, such as the ASCII code, from decimal to binary value, according to the predetermined value of 4, the number of digits obtained is 2. Another example: when the predetermined value is 8, the number of digits is 3; when the predetermined value is 16, the number of digits is 4. After converting the standard code to a binary value, take the number of digits of the binary value. The taking method can be from the low bit to the high bit, or from the high bit to the low bit, or taking the middle bit. Taking Chinese as an example, the decimal standard code corresponding to the character "Yang" is 26954. After converting it to binary representation as 110100101001010, and the predetermined value is 8, then the number of digits is 3. If taking 3 bits from the low bit to the high bit, that is, obtaining 010 as the representative code; if taking 3 bits from the high bit to the low bit, then obtaining 110 as the representative code; if taking the middle 3 bits, then obtaining 101 as the representative code. Finally, according to the representative code and through the lookup table, obtain the corresponding character. For example: if the representative code is 010, it can be looked up and corresponded to the character "N" with an index value of 2; if the representative code is 110, it can be looked up and corresponded to the character "E" with an index value of 6; if the representative code is 101, it can be looked up and corresponded to the character "Q" with an index value of 5. It should be noted that the same corresponding conversion method needs to adopt a fixed way of taking the representative code and the same lookup table.

[0059] Please refer to Figure 4In order to optimize and quickly compare according to the size of different file data, the high-repetition character comparison method of this embodiment can be used to achieve Figure 1 The illustrated step S3 includes the following steps. Through step S30, it is first determined whether the total number of the plurality of characters in the first intermediate file is greater than a first threshold. For example, the first threshold may be, but is not limited to, 8,000 characters. If the total number of characters is greater than the first threshold, then through step S31, a modified FM index method (Ferragina-Manzini index, FM-index) is executed to compare the second intermediate file with the first intermediate file to output a comparison result; otherwise, if the total number of characters is not greater than the first threshold, then through step S32, a table search method is executed to compare the second intermediate file with the first intermediate file to output a comparison result.

[0060] In some embodiments, the improved FM indexing method is based on the FM index data structure generated by the block sorting transformation (Burrows-Wheeler Transform, BWT), and detects and searches for characters in at least one first intermediate file with a large amount of data. For example, the first intermediate file includes a character sequence perspective, and an identification symbol $ is added to the character sequence and the character sequence is rotated in sequence to generate a plurality of rotation strings, as shown in Table 2 below. Then they are sorted according to the numerical value in the ASCII code to obtain a rotation table, as shown in Table 3 below.

[0061] Table 2

[0062] Original order Rotating string Original order Rotating string 0 perspective$ 6 ctive$perspe 1 erspective$p 7 tive$perspec 2 rspective$pe 8 ive$perspect 3 spective$per 9 ve$perspective 4 pective$pers 10 e$perspectiv 5 ective$persp 11 $perspective

[0063] Table 3 Rotary Table

[0064]

[0065]

[0066] Next, a counting table, a block-sorted compressed string, a sample table, and a location table are generated based on the rotation table, as shown in Tables 4 to 7 below. That is, the simplified FM index data structure of this embodiment includes a counting table, a block-sorted compressed string, a sample table, and a location table, which are suitable for decompressing the first intermediate file. Based on this, we can use the FM index data structure in the high-repetition character matching process to achieve the purpose of effectively compressing the intermediate file and quickly searching for characters, and it has the characteristic of being easily accelerated by hardware parallel computing.

[0067] Table 4 Counting table

[0068] character $ c E i p r s t v Sorting 0 1 2 5 6 8 9 10 11

[0069] Table 5 Sorting and Compressing Strings

[0070] e e v p P t s $ e r c i

[0071] Table 6 Sampling Table

[0072]

[0073]

[0074] Table 7 Position table

[0075] 11 5 4 3

[0076] Therefore, the improved FM index method is a high-volume data search method in a compressed state, which has the effect of compressing the data volume and searching quickly to save the memory required by the computer. In detail, according to the characters of the first intermediate file, a simplified FM index data structure is established through a table building step, and a plurality of adjacent characters of a predetermined length are sequentially intercepted (sliding sampling) for the character sequence of the second intermediate file to obtain a plurality of intercepted strings, and then, whether there is a character string in the character sequence of the first intermediate file that is the same as the aforementioned intercepted string is compared.

[0077] Please refer to Figure 5In this embodiment, the improved FM indexing method includes the following steps. First, through step S310, the first intermediate file is subjected to block sorting conversion to obtain an FM index data structure. The block sorting conversion example is described above. Secondly, through step S312, according to a predetermined length, such as but not limited to the character length consisting of 3 consecutive characters, the character sequence of the second intermediate file is sequentially intercepted (sliding sampling) to obtain adjacent intercepted strings in sequence. For example, the second intermediate file includes a character sequence ARGQNCCGAEAGGADDD. After sequential interception, a plurality of intercepted strings ARG, RGQ, GQN, ..., DDD can be obtained; then, each intercepted string is used to perform short word mapping (short read mapping) on ​​the FM index data structure to obtain a plurality of mapping results, for example: the mapping result is the first character hit interval to indicate the character string in the first intermediate file that is the same as the intercepted string from the second intermediate file. Then, through step S314, the plurality of mapping results are integrated into a comparison result. Take the above-mentioned predetermined value q=8 of different characters in the intermediate file as an example, if there are 10 mapping results arranged in a continuous position, all of which show that the intercepted string consisting of 3 consecutive characters in the second intermediate file corresponds to the same content in the first intermediate file, these 10 mapping results are integrated into the comparison result, which means that there are 12 consecutive characters that are the same in the first intermediate file, reaching the above-mentioned plagiarism threshold, and it can be determined that part of the content of the file to be checked is plagiarized from the existing file, as mentioned above. In some embodiments, the plagiarism threshold is greater than or equal to the predetermined length.

[0078] Please refer to Figure 6 In this embodiment, for the first intermediate file with a small number of characters, a highly repeated character comparison can be performed through a table search method. First, through step S320, q types of different characters are arranged and combined into a plurality of character keys. For example, the predetermined value of the different characters is q=8, and the predetermined length is 8 consecutive characters that are the same. Then, a total of 88=16777216 character keys are established in the file key value table, including key1, key2, ..., key16777216 and other preset fields corresponding to character sequences such as AAAAAAAA, AAAAAAAR, AAAAAAAN, ..., GGGGGGGG. Then, the character sequence of the first intermediate file is filled into the corresponding character key field in the file key value table according to 8 consecutive characters, as shown in Table 8 below, to establish a file key value table (HashMap1) for subsequent search and comparison, where doc12, doc201, ..., etc. represent the serial numbers of the plurality of first intermediate files.

[0079] Table 8

[0080] HashMap1={'key1':[doc12,doc201,…],

[0081] 'key2':[doc43,doc70,…],…

[0082] 'key16777216':[doc12,doc14,…]}

[0083] Through step S322, a position key table (HashMap2) is obtained according to the different characters and the first intermediate file, and the position of each character key in the first intermediate file is further recorded, as shown in Table 9 below.

[0084] Table 9

[0085] HashMap2={'key1':pos1,pos420,…],

[0086] 'key2':[pos4,…],……

[0087] 'key16777216':[pos 12,pos 14,…]}

[0088] Next, through step S324, the second intermediate file is sequentially intercepted (sliding sampling) according to the predetermined length, and the intercepted strings are sequentially obtained, and each intercepted string is searched in the file key table and the position key table to obtain the search result, for example: a plurality of intercepted strings are sequentially intercepted from the second intermediate file, and the file key table is compared to see whether there is a character key containing the same character string, and then one or more corresponding first intermediate files are downloaded to the temporary memory (RAM) according to the position key table, and the search is continued to output the character hit interval of the first intermediate file as the search result. Then, through step S326, a plurality of search results are integrated as the comparison result, for example: the character strings corresponding to the search results are integrated, and it is determined whether the number of characters meets the plagiarism threshold to output the comparison result.

[0089] In general, the table building and search method includes a table building step and a table lookup step. First, the table building step is performed: based on a plurality of different characters and the first intermediate file, a file key value table is obtained, the file key value table includes a plurality of preset fields for character keys, and the character sequence of the first intermediate file is sequentially intercepted and filled into the corresponding character key fields, and then, based on a plurality of different characters and the first intermediate file, a position key value table is obtained to record the position of each character key value in the first intermediate file. Secondly, the table lookup step is performed again: based on a predetermined length, such as but not limited to a character string consisting of 3 consecutive characters, the second intermediate file is sequentially intercepted, and the intercepted strings are sequentially obtained, and each intercepted string is searched in the file key value table and the position key value table to obtain the search result. Then, the plurality of search results are integrated into the above-mentioned comparison result. Taking the aforementioned predetermined value q=8 of different characters in the intermediate file as an example, if there are 10 consecutive search results that all show that the intercepted string consisting of 3 consecutive characters in the second intermediate file corresponds to the same content in the first intermediate file, these 10 search results are integrated into the comparison result, which means that there are 12 consecutive characters that are the same in the first intermediate file, reaching the above-mentioned plagiarism threshold. It can be inferred that part of the content of the file to be checked is plagiarized from the existing file, as mentioned above.

[0090] In summary of the above embodiments, the characters of the (first / second) intermediate file after irreversible conversion have a highly repetitive characteristic and cannot be restored from the intermediate file itself to the original file, which has the effect of high confidentiality and data compression.

[0091] In some embodiments, the step of receiving the second intermediate file includes but is not limited to: receiving the file to be inspected, and performing irreversible conversion on the file to be inspected to obtain the second intermediate file.

[0092] Please refer to Figure 3 , depicting a functional block diagram of a text comparison system 2 of some embodiments of the present invention. The terminal device 1 is but not limited to a mobile phone, a tablet, a desktop computer, a laptop computer, or a client in a server system. The text comparison system 2 is but not limited to a server in a server system, a mobile phone, a desktop computer, a laptop computer, etc. In some embodiments, the text comparison system 2 is a text comparison server. The communication connection is but not limited to wired communication or wireless communication, wired communication is for example but not limited to: twisted pair, coaxial cable and optical fiber, etc., and wireless communication is for example but not limited to: wireless local area network (WIFI), cellular network (3G, 4G), Zigbee, NFC, Bluetooth, etc. The text comparison system includes a terminal device 1 located at the local end and a text comparison system 2 located at the server end. There is a communication connection between the terminal device 1 and the text comparison system 2.

[0093] In some embodiments, Figure 1 The text comparison method can be achieved by Figure 3 . For example, the user can transmit the existing document and the document to be checked from the terminal device 1 to the text comparison system 2, and the text comparison system 2 executes the text comparison method. After the text comparison system 2 compares, the comparison result is returned to the terminal device 1. The user can know whether the document to be checked has part of the same content as the existing document through the comparison result. This implementation mode can be called a pairwise inspection mode. In some embodiments, the user transmits the document to be checked from the terminal device 1 to the text comparison system 2, and the text comparison system 2 performs a text comparison between the document to be checked and the existing document stored in the text comparison system 2. After the text comparison system 2 compares, the comparison result is returned to the terminal device 1. In some embodiments, the user transmits the document to be checked from the terminal device 1 to the text comparison system 2, and the text comparison system 2 obtains public data as the existing document through the Internet search, and compares the document to be checked. After the text comparison system 2 compares, the comparison result is returned to the terminal device 1. This implementation mode can be called an Internet inspection mode.

[0094] Please refer to Figure 1 and Figure 7 , Figure 1 The text comparison method can be achieved by Figure 7 Implementation of text comparison system 2. Figure 7 A circuit block diagram of a text comparison system 2 according to some embodiments of the present invention is shown. In this embodiment, the text comparison method is implemented by a computer program, so that when a computer (i.e., any electronic device including the computing unit 20 and the communication module 22, such as the text comparison system 2) loads the program and executes it, the text comparison method of any embodiment can be completed.

[0095] As mentioned above, the text matching system 2 can be any electronic device including a computing unit 20 and a communication module 22. In some embodiments, the communication module 22 can be a wireless communication interface, which establishes a connection with the terminal device 1 from the user through a wireless communication protocol, and the wireless communication protocol has been described above. In other embodiments, the communication module 22 can be a wired communication interface, which can be used for example, but not limited to, twisted pair, cable, and optical fiber transmission methods to establish a connection with the terminal device 1, as described above.

[0096] The computing unit 20 receives the second intermediate file through the communication module 22 , wherein the relevant technical contents and functions of the receiving mechanism and the second intermediate file are as described above.

[0097] The operation unit 20 is electrically connected to the communication module 22. In one embodiment, the operation unit 20 can be implemented by one or more processing components such as a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit and / or any processing component based on an operation instruction operation signal (analog and / or digital). The operation unit 20 performs an irreversible conversion on the existing file to obtain a first intermediate file, wherein the relevant calculation mechanism and the derivative embodiments of the irreversible conversion and the first intermediate file are as described above.

[0098] The operation unit 20 compares the second intermediate file with the first intermediate file using a high-repetition character comparison method according to a predetermined length to output a comparison result. In some embodiments, the operation unit 20 determines whether the total number of the plurality of characters in the first intermediate file is greater than a first threshold, and accordingly performs an improved FM index method or a table search method to compare the second intermediate file with the first intermediate file to output a comparison result. How the operation unit 20 performs the high-repetition character comparison mechanism and its derivative embodiments have been described above.

[0099] In at least one embodiment, the file comparison system 2 may selectively include a storage unit 24. The storage unit 24 is electrically connected to the computing unit 20. In one embodiment, the storage unit 24 may be implemented by one or more memories. The storage unit 24 may store a plurality of intermediate files, such as a first intermediate file and a second intermediate file, or the storage unit 24 may store at least one file that can be converted into a corresponding first intermediate file.

[0100] In some embodiments, the operation unit 20 is adapted to perform irreversible conversion. The operation unit 20 converts each character of the corresponding file into one of a plurality of different characters, and integrates a plurality of corresponding characters to output a first intermediate file, wherein the operation unit 20 obtains a standard code corresponding to the character; obtains a remainder of the standard code divided by a predetermined value; and obtains a corresponding character in a lookup table according to the remainder to implement the aforementioned corresponding conversion mechanism.

[0101] In some embodiments, the operation unit 20 establishes a lookup table including index values ​​and a plurality of different characters corresponding to respective index values, and the index value is an integer from zero to a value less than a predetermined value.

[0102] In some embodiments, the function of the operation unit 20 to implement the highly repeated matching characters has been described above and will not be described in detail here.

[0103] In some embodiments, the computer program product for the text comparison method is composed of a set of instructions. When a computer loads and executes the set of instructions, the text comparison method of any of the above embodiments can be completed.

[0104] In summary, some embodiments of the present invention provide a text comparison method, system and computer program product thereof, which mainly utilize irreversible encoding conversion to convert an original file containing human characters into an intermediate file containing a certain number of characters, wherein the intermediate file is converted by, for example, a many-to-one function relationship algorithm, so that the multiple characters contained in the intermediate file have a highly repetitive characteristic, and the original file cannot be deciphered, and has the effect of high confidentiality and highly compressed data; at the same time, the high repetitive character comparison method is used to perform plagiarism detection on the above intermediate file to determine whether there is plagiarism. When the total number of characters in the first intermediate file is large, the improved FM index method is executed, which can greatly reduce the memory usage required during the establishment of the FM index data structure, and can effectively achieve the effect of compressing huge amounts of data and searching for characters at a relatively low hardware cost and relatively short running time. When the total number of characters in the first intermediate file is small, the table building search method is executed, which can also quickly search for plagiarized content, and has many advantages as mentioned above.

[0105] The embodiments described above are only for illustrating the technical ideas and features of the present invention, and their purpose is to enable people skilled in the art to understand the contents of the present invention and implement them accordingly. They cannot be used to limit the patent scope of the present invention. That is, all equivalent changes or modifications made according to the spirit disclosed by the present invention should still be included in the patent scope of the present invention.

[0106]

Explanation of symbols

[0107] S1~S3、S10~S14、

[0108] Steps S30~S32, S310~S314, S320~S326

[0109] 1 Terminal device

[0110] 2 Text matching system

[0111] 20 Arithmetic Units

[0112] 22 Communication Module

[0113] 24 storage units.

Claims

1. A text comparison method, suitable for comparing a document to be inspected with an existing document, the comparison method comprising: Performing irreversible conversion on the existing file to obtain a first intermediate file, the first intermediate file comprising a plurality of characters, the number of the different characters being a predetermined value; Receiving a second intermediate file, wherein the second intermediate file is a file obtained by irreversibly converting the file to be inspected; as well as Outputting a comparison result by sequentially comparing character strings of a predetermined length according to the second intermediate file and the first intermediate file; The step of outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Performing block sorting conversion on the first intermediate file to obtain an FM index data structure; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is used to perform short word mapping on the FM index data structure to obtain a plurality of mapping results; and The mapping results are integrated into the comparison result.

2. A text comparison method, suitable for comparing a document to be inspected with an existing document, the comparison method comprising: Performing irreversible conversion on the existing file to obtain a first intermediate file, the first intermediate file comprising a plurality of characters, the number of the different characters being a predetermined value; Receiving a second intermediate file, wherein the second intermediate file is a file obtained by irreversibly converting the file to be inspected; as well as Outputting a comparison result by sequentially comparing character strings of a predetermined length according to the second intermediate file and the first intermediate file; The step of outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Obtaining a file key value table according to the different characters and the first intermediate file; Obtaining a position key table according to the different characters and the first intermediate file; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is searched in the file key table and the position key table to obtain a search result; and The search results are integrated into the comparison result.

3. A text comparison method, suitable for comparing a document to be inspected with an existing document, the comparison method comprising: Performing irreversible conversion on the existing file to obtain a first intermediate file, the first intermediate file comprising a plurality of characters, the number of the different characters being a predetermined value; Receiving a second intermediate file, wherein the second intermediate file is a file obtained by irreversibly converting the file to be inspected; as well as Outputting a comparison result by sequentially comparing character strings of a predetermined length according to the second intermediate file and the first intermediate file; The step of outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Determine whether the total number of the characters in the first intermediate file is greater than a first threshold, and if it is greater than the first threshold, execute: Performing block sorting conversion on the first intermediate file to obtain an FM index data structure; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is used to perform short word mapping on the FM index data structure to obtain a plurality of mapping results; and Integrating the mapping results into the comparison result; and Determine whether the total number of the characters in the first intermediate file is greater than a first threshold, and if not greater than the first threshold, execute: Obtaining a file key value table according to the different characters and the first intermediate file; Obtaining a position key table according to the different characters and the first intermediate file; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is searched in the file key table and the position key table to obtain a search result; and The search results are integrated into the comparison result.

4. The method of any one of claims 1 to 3, wherein the irreversible conversion comprises: Correspondingly converting each character of the existing document into one of the different characters, and integrating the corresponding characters to output the first intermediate document, wherein the corresponding conversion includes: Get the standard encoding corresponding to the character; obtaining a remainder of dividing the standard code by the predetermined value; and According to the remainder, the corresponding character is obtained in a lookup table.

5. The method of claim 4, wherein the lookup table comprises index values ​​and the different characters corresponding to each index value, and the index value is an integer from zero to less than the predetermined value.

6. The method of any one of claims 1 and 3, wherein the FM index data structure comprises a count table, a block-ordered compressed string, a sample table, and a position table.

7. The method according to any one of claims 1 to 3, wherein the step of receiving the second intermediate file comprises: receiving the document to be inspected; as well as The irreversible conversion is performed on the file to be inspected to obtain the second intermediate file.

8. The method as claimed in any one of claims 1 to 3, wherein the existing file also obtains a position index file after the irreversible conversion, which is used to correspond each character position of the first intermediate file to each character position of the existing file.

9. The method as claimed in any one of claims 1 to 3, wherein the comparison result comprises at least one of a first character hit interval of the first intermediate file and a second character hit interval of the second intermediate file.

10. The method of any one of claims 1 to 3, wherein the irreversible conversion comprises: Correspondingly converting each character of the existing document into one of the different characters, and integrating the corresponding characters to output the first intermediate document, wherein the corresponding conversion includes: According to the predetermined value, a number of digits is obtained; Obtain a representative code according to the number of digits and the standard code; and According to the representative code, the corresponding character is obtained in a lookup table.

11. A computer program product, comprising a set of instructions, which can implement the method as claimed in any one of claims 1 to 3 when loaded and executed by a computer.

12. A file comparison system, suitable for comparing a file to be inspected with an existing file, the file comparison system comprising: A computing unit is used for performing irreversible conversion on the existing file to obtain a first intermediate file, wherein the first intermediate file includes a plurality of characters, and the number of the different characters is a predetermined value, and the computing unit outputs a comparison result by sequentially comparing a character string of a predetermined length with the first intermediate file according to the second intermediate file; as well as A communication module, electrically connected to the computing unit, wherein the computing unit receives the second intermediate file through the communication module, and the second intermediate file is a file of the file to be inspected after the irreversible conversion; in, The step of the operation unit outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Performing block sorting conversion on the first intermediate file to obtain an FM index data structure; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is used to perform short word mapping on the FM index data structure to obtain a plurality of mapping results; and The mapping results are integrated into the comparison result.

13. A file comparison system, suitable for comparing a file to be inspected with an existing file, the file comparison system comprising: A computing unit is used for performing irreversible conversion on the existing file to obtain a first intermediate file, wherein the first intermediate file includes a plurality of characters, and the number of the different characters is a predetermined value, and the computing unit outputs a comparison result by sequentially comparing a character string of a predetermined length with the first intermediate file according to the second intermediate file; as well as A communication module, electrically connected to the computing unit, wherein the computing unit receives the second intermediate file through the communication module, and the second intermediate file is a file of the file to be inspected after the irreversible conversion; in, The step of the operation unit outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Obtaining a file key value table according to the different characters and the first intermediate file; Obtaining a position key table according to the different characters and the first intermediate file; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is searched in the file key table and the position key table to obtain a search result; and The search results are integrated into the comparison result.

14. A file comparison system, suitable for comparing a file to be inspected with an existing file, the file comparison system comprising: A computing unit is used for performing irreversible conversion on the existing file to obtain a first intermediate file, wherein the first intermediate file includes a plurality of characters, and the number of the different characters is a predetermined value, and the computing unit outputs a comparison result by sequentially comparing a character string of a predetermined length with the first intermediate file according to the second intermediate file; as well as A communication module, electrically connected to the computing unit, wherein the computing unit receives the second intermediate file through the communication module, and the second intermediate file is a file of the file to be inspected after the irreversible conversion; in, The step of the operation unit outputting the comparison result by sequentially comparing the character string of the predetermined length according to the second intermediate file and the first intermediate file includes: Determine whether the total number of the characters in the first intermediate file is greater than a first threshold, and if it is greater than the first threshold, execute: Performing block sorting conversion on the first intermediate file to obtain an FM index data structure; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is used to perform short word mapping on the FM index data structure to obtain a plurality of mapping results; and Integrating the mapping results into the comparison result; and Determine whether the total number of the characters in the first intermediate file is greater than a first threshold, and if not greater than the first threshold, execute: Obtaining a file key value table according to the different characters and the first intermediate file; Obtaining a position key table according to the different characters and the first intermediate file; According to the predetermined length, the second intermediate file is sequentially intercepted to obtain intercepted strings in sequence, and each of the intercepted strings is searched in the file key table and the position key table to obtain a search result; and The search results are integrated into the comparison result.

Images