File comparison method and device, electronic equipment and storage medium

Abstract

The invention discloses a file comparison method and device, electronic equipment and a storage medium, and belongs to the technical field of text processing, in the method, a computing node receives a first text block and a second text block sent by a control node, and the block division order of the first text block in a first file is the same as the block division order of the second text block in a second file; carrying out similarities and differences labeling on characters in the first text block and the second text block, and carrying out iterative calculation on resource consumption conditions of the first text block and the second text block under different character comparison paths according to similarities and differences labeling results and a resource consumption iteration rule during character comparison, according to an iterative calculation result and character comparison rules corresponding to different path backtracking modes, a block comparison result is determined, the block comparison result comprises character modification operation performed when the first text block is updated into the second text block, the block comparison result is sent to a control node, and the control node controls the second text block according to the block comparison result. And determining a file comparison result of the first file and the second file.

Description

Technical Field

[0001] This application relates to the technical field of text processing, and in particular, to a file comparison method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of information technology and the rapid growth of data scale, the need for content comparison of files has become increasingly widespread, especially in fields such as academia, law, business, and data management.

[0003] The file comparison methods in the related art focus on calculating the overall similarity of files, and it is difficult to compare the specific differences between two files in a fine-grained manner. How to efficiently compare the specific differences between two files has become a technical problem to be solved urgently. Summary of the Invention

[0004] Embodiments of this application provide a file comparison method, apparatus, electronic device, and storage medium, so as to provide a solution for efficiently comparing the specific differences between two files.

[0005] In a first aspect, an embodiment of this application provides a file comparison method, including:

[0006] A computing node receives comparison data sent by a control node, where the comparison data includes a first text block and a second text block, and the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file;

[0007] Perform identical and different marking on the characters in the first text block and the second text block;

[0008] According to the identical and different marking results and the resource consumption iteration rule during character comparison, perform iterative calculation on the resource consumption situations of the first text block and the second text block under different character comparison paths;

[0009] According to the iterative calculation results and the character comparison rules corresponding to different path backtracking methods, determine the block comparison result, where the block comparison result includes the character modification operations for updating the first text block to the second text block;

[0010] Send the block comparison result to the control node, and trigger the control node to determine the file comparison result of the first file and the second file according to the block comparison results sent by multiple computing nodes.

[0011] In a second aspect, an embodiment of this application provides a file comparison method, including:

[0012] The control node divides the first file and the second file to be compared into the same number of text blocks;

[0013] Send comparison data to multiple computing nodes, where the comparison data sent to each computing node includes a first text block and a second text block, and the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file. The comparison data is used to trigger the computing node to perform similarity and difference annotation on the characters in the first text block and the second text block, and iteratively calculate the resource consumption when comparing the characters in the first text block and the second text block according to the similarity and difference annotation results and the resource consumption iteration rule during character comparison; determine the block comparison result according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods during comparison. The block comparison result includes the character modification operations for updating the first text block to the second text block.

[0014] Receive the block comparison results sent by the multiple computing nodes.

[0015] Generate the file comparison result of the first file and the second file according to each block comparison result.

[0016] In a third aspect, an embodiment of the present application provides a file comparison device, including:

[0017] A receiving module, configured to receive comparison data sent by a control node, where the comparison data includes a first text block and a second text block, and the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file;

[0018] An annotation module, configured to perform similarity and difference annotation on the characters in the first text block and the second text block;

[0019] A consumption determination module, configured to iteratively calculate the resource consumption when comparing the characters in the first text block and the second text block according to the similarity and difference annotation results and the resource consumption iteration rule during character comparison;

[0020] A comparison module, configured to determine the block comparison result according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods during comparison. The block comparison result includes the character modification operations for updating the first text block to the second text block;

[0021] A sending module, configured to send the block comparison result to the control node, and the control node determines the file comparison result of the first file and the second file according to the block comparison result.

[0022] In a fourth aspect, an embodiment of the present application provides a file comparison device, including:

[0023] A division module, configured to divide a first file and a second file to be compared into the same number of text blocks;

[0024] A sending module, configured to send comparison data to multiple computing nodes. The comparison data sent to each computing node includes a first text block and a second text block. The block division order of the first text block in the first file is the same as the block division order of the second text block in the second file. The comparison data is used to trigger the computing node to perform same / different character annotation on the characters in the first text block and the second text block, and perform iterative calculation on the resource consumption when comparing the characters in the first text block and the second text block according to the same / different annotation results and the resource consumption iteration rule during character comparison; according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods during comparison, determine the block comparison result, where the block comparison result includes the character modification operation for updating the first text block to the second text block.

[0025] A receiving module, configured to receive the block comparison results sent by the multiple computing nodes.

[0026] A generating module, configured to generate a file comparison result of the first file and the second file according to each block comparison result.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, where:

[0028] The memory stores a computer program executable by the at least one processor. When the computer program is executed by the at least one processor, the at least one processor is enabled to execute any of the above file comparison methods.

[0029] In a sixth aspect, an embodiment of the present application provides a storage medium. When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above file comparison methods.

[0030] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements any of the above file comparison methods.

[0031] In an embodiment of the present application, a computing node receives comparison data sent by a control node. The comparison data includes a first text block and a second text block. The block division order of the first text block in the first file is the same as that of the second text block in the second file. Then, the characters in the first text block and the second text block are marked for similarities and differences. According to the result of the similarities and differences marking and the resource consumption iteration rule during character comparison, the resource consumption of the first text block and the second text block under different character comparison paths is iteratively calculated. Furthermore, according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods, the block comparison result is determined. The block comparison result includes the character modification operations for updating the first text block to the second text block, and the block comparison result is sent to the control node. The control node determines the file comparison result of the first file and the second file according to the block comparison result. In this way, multiple computing nodes can perform text block comparison respectively, which can improve the comparison speed and is suitable for large file comparison. Moreover, the character modification operations between text blocks can be compared, that is, the specific difference positions can be located, and the comparison granularity is finer. Description of the Drawings

[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0033] Figure 1 It is a flowchart of a file comparison method provided by an embodiment of the present application;

[0034] Figure 2 It is a flowchart of a method for iteratively calculating the resource consumption of a first text block and a second text block under different character comparison paths provided by an embodiment of the present application;

[0035] Figure 3 It is a flowchart of a method for determining a block comparison result provided by an embodiment of the present application;

[0036] Figure 4 It is a schematic structural diagram of a file comparison device provided by an embodiment of the present application;

[0037] Figure 5 It is a schematic structural diagram of another file comparison device provided by an embodiment of the present application;

[0038] Figure 6 It is a schematic hardware structure diagram of an electronic device for implementing the file comparison method provided by an embodiment of the present application. Detailed Embodiments

[0039] Embodiments of the present application provide a file comparison method, device, electronic device, and storage medium to efficiently compare the specific differences between two files.

[0040] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0041] Reference Figure 1 , Figure 1 is an interaction flowchart of a file comparison method provided by an embodiment of the present application, including the following steps.

[0042] In step 101, the control node divides the first file and the second file to be compared into the same number of text blocks.

[0043] Among them, both the first file and the second file are preprocessed files, the files have no format, and the files do not contain special characters, modal particles, etc. Generally, there is a certain similarity between the first file and the second file. For example, the second file is an upgraded version of the first file. In addition, both the first file and the second file can be large files (files with a size greater than a set value).

[0044] In specific implementation, when the similarity between the first file and the second file is low, such as lower than a preset value, a suitable block division strategy can be set according to the file size, such as 100,000 characters per block. When the similarity between the first file and the second file is high, such as not lower than the preset value or the file structure is relatively complex, the blocks can be divided according to natural paragraphs or chapters.

[0045] In step 102, the control node sends comparison data to multiple computing nodes.

[0046] Among them, the comparison data sent by the control node to each computing node may include the first text block and the second text block, and the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file.

[0047] In step 103, the computing node marks the similarities and differences of the characters in the first text block and the second text block in the comparison data.

[0048] For example, for the i-th character ai in the first text block and the j-th character bj in the second text block by the computing node: when ai and bj are the same, the similarity and difference marking value C[i][j] of ai and bj can be determined as a first preset value, such as 0; when ai and bj are different, the similarity and difference marking value C[i][j] of ai and bj can be determined as a second preset value, such as 1. Among them, 1≤i≤m, 1≤j≤n, m is the number of characters included in the first text block, and n is the number of characters included in the second text block.

[0049] In step 104, the computing node iteratively calculates the resource consumption of the first text block and the second text block under different character comparison paths according to the similarity and difference annotation results and the resource consumption iteration rule during character comparison.

[0050] Among them, the resource consumption during character comparison, such as the total number of character comparisons, the total character comparison time, etc., can reflect the resource consumption situation during character comparison. And each character comparison path represents a way of comparing the characters in the first text block and the second text block. Generally, in different character comparison paths, the comparison order of the characters in the first text block and the second text block is different.

[0051] Generally, when comparing the first character a1 in the first text block and the first character b1 in the second text block, the resource consumption value P[1][1] can be C[1][1]. Subsequently, any character comparison path uses P[1][1] as the initial value of the resource consumption.

[0052] During specific implementation, according to Figure 2 the shown process, iteratively calculate the resource consumption of the first text block and the second text block under different character comparison paths, including the following steps.

[0053] In step 1041, for ai and bj, select the smallest value from the cumulative values of the resource consumption of the first text block and the second text block under different character comparison paths as the reference cumulative value.

[0054] Among them, the cumulative values of the resource consumption of the first text block and the second text block under different character comparison paths may include: the cumulative value P[i - 1][j] of the resource consumption when comparing ai - 1 and bj, the cumulative value P[i][j - 1] of the resource consumption when comparing ai and bj - 1, and at least one of the cumulative value P[i - 1][j - 1] of the resource consumption when comparing ai - 1 and bj - 1. Here, ai - 1 is the (i - 1)th character in the first text block, and bj - 1 is the (j - 1)th character in the second text block.

[0055] It should be noted that generally, as many of P[i - 1][j], P[i][j - 1], and P[i - 1][j - 1] as there are will be selected as the cumulative values of the resource consumption.

[0056] In step 1042, according to the reference cumulative value and the similarity and difference annotation value C[i][j] of ai and bj, determine the cumulative value P[i][j] of the resource consumption when comparing ai and bj.

[0057] For example, P[i][j] = reference cumulative value + C[i][j].

[0058] In the above manner, the resource consumption of character comparison between the first text block and the second text block under different character comparison paths can be calculated.

[0059] In step 105, the computing node determines the block comparison result according to the iterative calculation result and the character comparison rules corresponding to different path backtracking methods.

[0060] Among them, each path backtracking method corresponds to a variation method of the character comparison position. And the path backtracking methods include: the character comparison position in the first text block retreats by one, and the character comparison position in the second text block remains unchanged; the character comparison position in the first text block retreats by one, and the character comparison position in the second text block retreats by one; and the character comparison position in the first text block remains unchanged, and the character comparison position in the second text block retreats by one. The block comparison result includes the character modification operations for updating the first text block to the second text block.

[0061] In specific implementation, it can be determined according to Figure 3 the shown process to determine the block comparison result, including the following steps.

[0062] In step 1051, the cumulative value P[m][n] of the resource consumption when comparing am and bn is used as the first cumulative value P[x][y].

[0063] In step 1052, the smallest value is selected from the previous cumulative value of the first cumulative value as the second cumulative value.

[0064] Among them, the previous cumulative value may include at least one of P[x - 1][y], P[x - 1][y - 1], and P[x][y - 1]. am is the mth character in the first text block, bn is the nth character in the second text block, P[x - 1][y] represents the cumulative value of the resource consumption when comparing ax - 1 and by, P[x - 1][y - 1] represents the cumulative value of the resource consumption when comparing ax - 1 and by - 1, and P[x - 1][y - 1] represents the cumulative value of the resource consumption when comparing ax and by - 1.

[0065] In some embodiments, when selecting the smallest value from the previous cumulative values of the first cumulative value as the second cumulative value, if there are at least two cumulative values with the smallest value in the previous cumulative values, and these at least two cumulative values with the smallest value include P[x - 1][y - 1], P[x - 1][y - 1] can be determined as the second cumulative value. This is because in various character comparison paths, advancing the character comparison positions in both the first text block and the second text block by one is the comparison method with the least resource consumption. Also because of this, when there are at least two cumulative values with the smallest value in the previous cumulative values, P[x - 1][y - 1] will be included in these at least two cumulative values with the smallest value, and there will be no situation where P[x - 1][y - 1] is not included.

[0066] In step 1053, according to the path backtracking method from the first cumulative value to the second cumulative value and the character comparison rule corresponding to the path backtracking method, determine the result of one character comparison.

[0067] That is, the character comparison result between the (x - 1)-th character ax - 1 in the first text block and the (y - 1)-th character by - 1 in the second text block.

[0068] Assume that the second cumulative value is P[x - 1][y]. Then, the path backtracking method from the first cumulative value P[x][y] to the second cumulative value P[x - 1][y] is: the character comparison position in the first text block retreats by one, and the character comparison position in the second text block remains unchanged. And the character comparison rule corresponding to this path backtracking method is: deletion operation. Then, it can be determined that the result of this character comparison is: delete ax - 1.

[0069] Assume that the second cumulative value is P[x - 1][y - 1]. Then, the path backtracking method from the first cumulative value P[x][y] to the second cumulative value P[x - 1][y - 1] is: the character comparison position in the first text block retreats by one, and the character comparison position in the second text block retreats by one. And the character comparison rule corresponding to this path backtracking method is: if C[x - 1][y - 1] is the second preset value, it is a replacement operation; if C[x - 1][y - 1] is the first preset value, the characters are the same. Then, it can be determined that the result of this character comparison is: if C[x - 1][y - 1] is the second preset value, replace ax - 1 with by - 1; if C[x - 1][y - 1] is the first preset value, ax - 1 and by - 1 are the same.

[0070] Assume that the second cumulative value is P[x][y - 1]. Then, when backtracking the path from the first cumulative value P[x][y] to the second cumulative value P[x][y - 1], the character comparison position in the first text block remains unchanged, and the character comparison position in the second text block moves back by one. Moreover, the character comparison rule corresponding to this path backtracking method is: insertion operation. Then, it can be determined that the result of this character comparison is: insert by-1.

[0071] In step 1054, it is determined whether there is a previous cumulative value for the first cumulative value. If so, step 1055 is entered; otherwise, step 1056 is entered.

[0072] In step 1055, the second cumulative value is used as the new first cumulative value, and step 1052 is returned.

[0073] In step 1056, the results of each character comparison are combined in order to obtain the block comparison result of the first text block and the second text block.

[0074] In some embodiments, the comparison data may further include the context of the first text block (i.e., the text before and after the first text block) and the context of the second text block (i.e., the text before and after the second text block).

[0075] In this case, after obtaining the block comparison result of the first text block and the second text block, it can also be determined whether the comparison result meets the re-comparison condition, such as whether the character modification operations are distributed in the head region (referring to the first preset proportion of characters in the second text block) and / or the tail region (referring to the last preset proportion of characters in the second text block) of the second text block.

[0076] If it meets the condition, it indicates that the block division of the second text block is not very appropriate, which may affect the accuracy of the final file comparison. Therefore, the first text block can be updated based on the context of the first text block. For example, the text before in the context is concatenated in front of the first text block, and the text after is concatenated behind the first text block. The second text block is updated based on the context of the second text block. Similarly, the text before in the context is concatenated in front of the second text block, and the text after is concatenated behind the second text block. Then, the updated first text block and second text block are re-compared until the obtained block comparison result does not meet the re-comparison condition.

[0077] In some embodiments, the block similarity between the first text block and the second text block can also be determined according to the number of identical characters between the first text block and the second text block and the total number of characters in the second text. For example, block similarity = 1 - (the number of identical characters between the first text block and the second text block / the total number of characters in the second text). And the block similarity can be added to the block comparison result.

[0078] In step 106, the computing node sends the block comparison result to the control node.

[0079] In step 107, the control node determines the file comparison result of the first file and the second file according to the block comparison results sent by multiple computing nodes.

[0080] In some embodiments, each block comparison result received by the control node further includes a block similarity, and the file similarity between the first file and the second file can also be determined according to each block similarity and the number of characters of each second text block.

[0081] For example, according to the following formula, calculate the file similarity:

[0082]

[0083] where S i is the block similarity of two text blocks with the block division order of i in the first file and the second file, and L i is the total number of characters of the i-th text block in the second file.

[0084] After that, merge the difference annotations of all block comparison results to generate a difference report of the first file and the second file. And the difference report can also include the file similarity between the first file and the second file, and the feature statistics of each text block in the first file and the second file, such as the word frequency distribution, the number and distribution of non-alphabetic characters (such as numbers, punctuation).

[0085] In this way, the control node divides the first file and the second file into the same number of text blocks, sends two text blocks with the same block division order to multiple computing nodes respectively, and the multiple computing nodes perform text block comparison in parallel. The comparison speed is relatively fast, which is suitable for large file comparison. And the computing node can compare the character modification operations between text blocks, that is, locate the specific difference positions, and the comparison granularity is also finer.

[0086] Next, taking two large files as an example, the file comparison process of the embodiments of the present application will be introduced.

[0087] Step 1. File preprocessing.

[0088] Input: Two large files to be compared.

[0089] Output: Two standardized and divided text blocks; the feature statistics, start and end positions, and context cache of each text block; the mapping table between the text block and the original file.

[0090] The specific processing process is as follows.

[0091] 1. File format parsing and unification.

[0092] Input: Two large files to be compared. The formats of the large files may be plain text files such as text files (TextFile, TXT), tokenized text such as HyperText Markup Language (HTML), eXtensible Markup Language (XML), Markdown, structured data such as JavaScript Object Notation (JSON), Comma-Separated Values (CSV), binary files such as Portable Document Format (PDF), Word documents, etc.

[0093] Output: Two standardized plain text files, and a mapping relationship table between the plain text files and the original large files (for subsequent difference location).

[0094] The operation steps are as follows:

[0095] 1.1 Detect the file types of the large files.

[0096] Generally, the file types can be judged according to the file extension or content features such as Magic Number. Among them, the file types are such as TXT files, HTML files, XML files, JSON files, PDF documents, Word documents, etc.

[0097] 1.2 Remove the format information in the large files.

[0098] For example, delete the tags and attributes in HTML files / XML files and retain the body content;

[0099] Filter the format symbols such as # and * in Markdown files;

[0100] Only retain the core field values in JSON files / CSV files.

[0101] 1.3 Extract the text data.

[0102] Convert the data in the large files into plain text strings, while retaining the original structure position information such as paragraph numbers or line numbers. Among them:

[0103] TXT files: No additional parsing is required, and the plain text content can be directly read;

[0104] HTML files / XML files: The corresponding parser can be used to extract the plain text content;

[0105] JSON file: Can extract keyword field values and concatenate them into continuous text;

[0106] PDF document / Word document: Can use Optical Character Recognition (OCR) or Application Programming Interface (API) such as PyPDF2, ApachePOI to extract the file content.

[0107] 2. Noise character cleaning.

[0108] Input: Standardized plain text file.

[0109] Output: Cleaned plain text file, with noise characters removed but core content retained.

[0110] The operation steps are as follows:

[0111] 2.1 Remove meaningless characters.

[0112] For example, delete extra spaces, line breaks, tab characters;

[0113] Remove duplicate punctuation that may exist in the text such as......,!!!!;

[0114] Filter out non-text symbols such as special delimiters and graphic symbols in the file.

[0115] 2.2 Unify character encoding.

[0116] Unify the character encoding of the file to UTF-8 to avoid character recognition errors caused by different encodings such as Chinese character garbled problems.

[0117] 2.3 Remove markup symbols.

[0118] For certain documents such as programming code or log files, delete irrelevant markup symbols such as comments and timestamps.

[0119] 2.4 Handle case issues.

[0120] If the comparison content is case-insensitive, convert it to lowercase uniformly; otherwise, retain the original case information.

[0121] 3. Chunk processing.

[0122] Input: Cleaned plain text file.

[0123] Output: Small files (i.e., text chunks) after chunking, along with their start position information and end position information;

[0124] The operation steps are as follows:

[0125] 3.1 Determine the block size.

[0126] When the file similarity is low, such as below the preset value, an appropriate block strategy can be set according to the file size, such as 100,000 characters per block.

[0127] When the file similarity is high, such as not lower than the preset value, or when the file structure is relatively complex, the blocks can be divided according to natural paragraphs or chapters.

[0128] 3.2 Slide window block division.

[0129] To reduce the block boundary error, an overlapping range between blocks can be set (for example, the last 10% of the characters in each block are repeated with the first 10% of the characters in the next block), record the start position and end position of each block (i.e., the text block), and retain this information in the block division.

[0130] It should be noted that the data between blocks does not overlap, but only context information for each block is retained.

[0131] 3.3 Feature statistics.

[0132] While dividing the blocks, generate basic statistical information for each block, such as:

[0133] Word frequency distribution: Record the occurrence frequency of common words.

[0134] Special character count: Record the quantity and distribution of non-alphabetic characters such as numbers and punctuation marks.

[0135] 4. Verification of the block division result.

[0136] Input: The file data after block division.

[0137] Output: Complete and correct block data, the mapping table between the original file and the block data.

[0138] The operation steps are as follows:

[0139] 4.1 Verify the block integrity: Ensure that all blocks cover all the content of the original file; check whether the overlapping parts between blocks are correct (to avoid missing boundary data).

[0140] 4.2 Record the block mapping relationship: For each block, record its line number, paragraph number, or character range in the original file; ensure that the comparison result can be traced back to the specific location of the original file.

[0141] 5. Boundary processing for file block division preparation.

[0142] Input: The file blocks after cleaning.

[0143] Output: The context cache of the block data; the block information of the marked low-sensitivity blocks.

[0144] The operation steps are as follows:

[0145] 5.1 Context splicing preprocessing:

[0146] For each block, extract context information (the last 10% of the previous block and the first 10% of the next block characters), and temporarily cache the context information for splicing during subsequent similarity calculation.

[0147] 5.2 Mark low-sensitivity blocks:

[0148] For some blocks with low similarity (such as the blocks at the beginning or end of two large files), markers indicating skipping comparison can be set to indicate that these blocks are low-sensitivity blocks. Subsequently, when the control node sees the marker, it can skip the comparison of these blocks and give priority to comparing other blocks.

[0149] After preprocessing, the control node can send two blocks and the context of these two blocks to each computing node. The block division order of these two blocks in the original two large files is the same.

[0150] Step 2: Compare the two blocks.

[0151] Mainly through the character comparison matrix C and the path similarity matrix P, combined with dynamic programming, accurately compare the character differences between large files. The specific process is as follows.

[0152] 1. Construct the character comparison matrix C.

[0153] 1.1 Matrix description and function.

[0154] The character comparison matrix C is a two-dimensional array used to store the character differences between two blocks such as block 1 and block 2.

[0155] Size: C is an m×n matrix, where m and n are the lengths (i.e., the total number of characters) of block 1 and block B respectively;

[0156] Meaning: C[i][j] represents the annotation result of the similarity or difference between the i-th character a i of block 1 and the j-th character b j of block 2.

[0157] 1.2 Mathematical representation.

[0158]

[0159] Among them, a i represents the i-th character in block 1; b j represents the j-th character in block 2.

[0160] 1.3 Example illustration.

[0161] Assume the content of block 1 is ABCDE and the content of block 2 is FABSD. Then the generated character comparison matrix C is as follows:

[0162] Character comparison matrix C B1(F) B2(A) B3(B) B4(S) B(5)D A1(A) 1 0 1 1 1 A2(B) 1 1 0 1 1 A3(C) 1 1 1 1 1 A4(D) 1 1 1 1 0 A5(E) 1 1 1 1 1

[0163] 2. Generate the path similarity matrix P.

[0164] 2.1 Matrix description and function.

[0165] The path similarity matrix P is used to record the minimum path cost from the upper left corner of the matrix to any position (i.e., the resource consumption of block 1 and block 2 under different character comparison paths), and is used to find the most suitable comparison method between block 1 and block 2.

[0166] 2.2 Mathematical representation.

[0167] Dynamic programming recurrence formula:

[0168]

[0169]

[0170] Among them, P[i][j] is the cost of comparing the i-th character ai in block 1 with the j-th character bj in block 2 (i.e., the cumulative value of resource consumption).

[0171] 2.3 Initial conditions.

[0172] P[1][1] = C[1][1], and subsequent values are obtained completely according to the dynamic programming formula.

[0173] 2.4 Example path similarity matrix P.

[0174] Based on the character comparison matrix C, the path similarity matrix P is calculated as follows:

[0175] Path similarity matrix P B1(F) B2(A) B3(B) B4(S) B(5)D A1(A) 1 1 2 3 4 A2(B) 2 2 1 2 3 A3(C) 3 3 2 2 3 A4(D) 4 4 3 3 2 A5(E) 5 5 4 4 3

[0176] 3. Backtrack to mark the local difference path.

[0177] Backtrack from the position P[m][n] at the lower right corner of the P matrix to find the shortest path and mark the local differences. Since the cost obtained for the corresponding position from the P matrix is the minimum, a conclusion can be drawn: the cost paid after the final comparison is P[m][n].

[0178] In this example, P[m][n] is P[5][5]. Then, backtrack from P[5][5]. The backtracking process is as follows.

[0179] P[5][5] = C[5][5] + min(P[5][4], P[4][4], P[4][5]). Among them, P[4][5] is the smallest. So the path selected in the previous step is P[4][5], and the selection method is the deletion operation, that is, moving from the position of P[4][5] to the position of P[5][5]. Block A needs to delete the character E relative to Block B;

[0180] P[4][5] = C[4][5] + min(P[4][4], P[3][5], P[3][4]). Among them, P[3][4] is the smallest. So the path selected in the previous step is P[3][4]. Since C[4][5] is equal to 0, the character D in Block 1 and Block 2 is the same;

[0181] P[3][4] = C[3][4] + min(P[2][4], P[3][3], P[2][3]). Among them, P[2][3] is the smallest. So the path selected in the previous step is P[2][3]. Since C[3][4] = 1, this step is a replacement operation. Block 1 needs to replace the character C with the character S relative to Block 2;

[0182] P[2][3] = C[2][3] + min(P[2][2], P[1][3], P[1][2]). Among them, P[1][2] is the smallest. So the path selected in the previous step is P[1][2]. Since C[2][3] is equal to 0, the character B in Block 1 and Block 2 is the same;

[0183] Since P[1][2] has compared the first character in Block 1 and the second character in Block 2, the previous step can only be P[1][1] (that is, comparing the first character in Block 1 and the first character in Block 2). Since C[1][2] = 0, the character A in Block 1 and Block 2 is the same;

[0184] P[1][1] is the starting position. Since C[1][1] = 1, Block 1 needs to insert the character F relative to Block 2.

[0185] In summary, the same characters in Block 1 and Block 2 are A, B, and D. The differences are that a character F is inserted at the beginning of Block 1, the middle character C is replaced with the character S, and the ending character E is deleted. And the similarity between Block 1 and Block 2 = 3 / 5 = 0.6.

[0186] It should be noted that when backtracking, if the costs on different paths are the same, the path that reduces both horizontally and vertically by 1 is preferentially selected (that is, the path backtracking method is that the character comparison positions in both Block 1 and Block 2 move back one). Because the number of comparison points on this path is the least, the path with the least number of comparison points and the smallest cost is the optimal character comparison path.

[0187] In practical applications, after obtaining the block comparison results of two blocks, if it is determined that the block comparison results do not meet the re-comparison conditions, such as the differences in the blocks being concentrated at the beginning and end, the block similarity can be calculated. For example, block similarity = 1 - (the number of identical characters in the two blocks / the total number of characters in the blocks being compared), and the block similarity is added to the block comparison results. If it is determined that the block comparison results meet the re-comparison conditions, then step three can be entered.

[0188] Step three: Concatenate the context of the blocks and re-perform the comparison.

[0189] The main purpose of context concatenation and correction is to ensure that when processing large files in chunks, misjudgments caused by chunk boundaries are corrected. Especially when the differences or similar content of the chunks cross the chunk boundaries, by concatenating the context of the chunks onto the chunks and recalculating the similarity, the accuracy of the final comparison can be maximally improved.

[0190] 1. Problem background.

[0191] When chunking a large file, the similarity calculation may lose the complete context due to the truncation of the large file.

[0192] For example, a large file A is chunked into multiple chunks, and two of the chunks are:

[0193] Chunk A1:...XYZ;

[0194] Chunk A2: ABC...

[0195] Another large file B is also chunked into multiple chunks, and the two chunks with the same partitioning order as chunk A1 and chunk A2 are:

[0196] Chunk B1:...XY;

[0197] Chunk B2: ZABC...

[0198] If only comparing by chunks separately:

[0199] Chunk A1 and chunk B1: The differences are large;

[0200] Chunk A2 and chunk B2: The differences are small.

[0201] In actual situations, XYZ and ZABC may be continuously matched, but are cut off by chunking, resulting in a distortion of the similarity. Therefore, it is necessary to concatenate the context at the chunk boundary and recalculate.

[0202] 2 Context concatenation and trimming process.

[0203] 2.1 Example of concatenating chunks.

[0204] Concatenate the context of block A2 to block A2. Assume the concatenation result is:...XYZABC. Concatenate the context of block B2 to block B2. Assume the concatenation result is: XYZABC....

[0205] 2.2 Recalculate the similarity.

[0206] Re - compare the concatenated block A2 and block B2 according to step two.

[0207] The comparison result of the re - compared blocks can still determine whether it meets the re - comparison condition. If it meets, the context of the current block can be requested from the control node again, and step three can be executed again until the block comparison result does not meet the re - comparison condition, and then the block comparison result is sent to the control node.

[0208] Step four: Result integration.

[0209] 1 Global similarity integration.

[0210] Calculate the file similarity S of two large files by weighted calculation according to the proportion of the number of characters in each block total :

[0211]

[0212] where S i is the block similarity of two blocks with the division order of i.

[0213] 2 Merge the difference information of all blocks to obtain the global difference information.

[0214] Step five: Result output and display.

[0215] 1. Visual output

[0216] Input: File similarity and global difference information.

[0217] Operation:

[0218] Generate graphical displays of similarity such as heat maps and difference distribution maps;

[0219] Highlight and mark the specific difference positions in the two large files.

[0220] Output: User - friendly visual report.

[0221] 2. Generation of file comparison report.

[0222] Input: File similarity and global difference information.

[0223] Operation:

[0224] Generate a structured comparison report, including similarity scores, main difference segments and their positions.

[0225] Export the report as PDF, Excel or other formats.

[0226] Output: A complete file comparison report.

[0227] In the embodiments of the present application, the control node divides two large files to be compared into blocks, and according to the rule of sending two blocks with the same block division order to one computing node, the divided blocks are sent to multiple computing nodes. The multiple computing nodes perform block comparison in parallel, discover the specific character differences between the blocks, and when differences are found at the beginning or end of the block, update the block quickly through context splicing and re-compare to ensure the accuracy of the block comparison result. Finally, the control node synthesizes the block comparison results of multiple computing nodes to obtain the final file comparison result. In this way, the distributed parallel comparison method is relatively fast, suitable for large file comparison, and can compare the character modification operations between text blocks, that is, locate the specific difference positions, and the comparison granularity is finer.

[0228] Based on the same technical concept, the embodiments of the present application also provide a file comparison device. The principle of the file comparison device to solve problems is similar to the above file comparison method. Therefore, for the implementation of the file comparison device, reference can be made to the implementation of the file comparison method, and the repeated parts will not be elaborated.

[0229] Figure 4 It is a schematic structural diagram of a file comparison device provided by an embodiment of the present application, including:

[0230] A receiving module 401, configured to receive comparison data sent by a control node, where the comparison data includes a first text block and a second text block, and the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file;

[0231] A marking module 402, configured to mark the similarities and differences of characters in the first text block and the second text block;

[0232] A consumption determination module 403, configured to perform iterative calculation on the resource consumption when comparing characters in the first text block and the second text block according to the similarity and difference marking results and the resource consumption iteration rule during character comparison;

[0233] A comparison module 404, configured to determine a block comparison result according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods during comparison, where the block comparison result includes the character modification operations for updating the first text block to the second text block;

[0234] A sending module 405, configured to send the block comparison result to the control node, and the control node determines the file comparison result of the first file and the second file according to the block comparison result.

[0235] In some embodiments, the annotation module 402 is specifically configured to:

[0236] For the i-th character ai in the first text block and the j-th character bj in the second text block: when ai and bj are the same, determine that the similarity and difference annotation value C[i][j] of ai and bj is a first preset value; when ai and bj are different, determine that the similarity and difference annotation value C[i][j] of ai and bj is a second preset value;

[0237] Where 1 ≤ i ≤ m, 1 ≤ j ≤ n, m is the number of characters included in the first text block, and n is the number of characters included in the second text block.

[0238] In some embodiments, the consumption determination module 403 is specifically configured to:

[0239] For ai and bj, select the smallest value from the cumulative values of resource consumption under different character comparison paths of the first text block and the second text block as the reference cumulative value;

[0240] According to the reference cumulative value and the similarity and difference annotation value C[i][j] of ai and bj, determine the cumulative value P[i][j] of resource consumption when comparing ai and bj.

[0241] In some embodiments, the first preset value is less than the second preset value, and the comparison module 404 is specifically configured to:

[0242] Take P[m][n] as the first cumulative value P[x][y], and select the smallest value from the previous cumulative values of the first cumulative value as the second cumulative value, where the previous cumulative values include at least one of P[x - 1][y], P[x - 1][y - 1], and P[x][y - 1];

[0243] According to the path backtracking method from the first cumulative value to the second cumulative value and the character comparison rule corresponding to the path backtracking method, determine the result of a single character comparison;

[0244] Take the second cumulative value as the new first cumulative value, and execute the step of selecting the smallest value from the previous cumulative values of the first cumulative value as the second cumulative value until there is no previous cumulative value for the first cumulative value, and then combine the results of each character comparison to obtain the block comparison result.

[0245] In some embodiments, the comparison module 404 is specifically configured to:

[0246] If there are at least two cumulative values with the smallest values among the previous cumulative values, and the at least two cumulative values with the smallest values include P[x - 1][y - 1], then P[x - 1][y - 1] is determined as the second cumulative value.

[0247] In some embodiments, the comparison data further includes the context of the first text block and the context of the second text block, and further includes a correction module 406 for:

[0248] The comparison data further includes the context of the first text block and the context of the second text block. If it is determined that the block comparison result meets the re - comparison condition, then based on the context of the first text block, the first text block is updated, and based on the context of the second text block, the second text block is updated;

[0249] For the updated first text block and second text block, perform the step of marking the similarities and differences of the characters in the first text block and the second text block until the obtained block comparison result does not meet the re - comparison condition.

[0250] In some embodiments, the re - comparison condition includes that the block comparison result indicates that the character modification operations are distributed in the head region and / or tail region of the second text block.

[0251] Figure 5 It is a schematic structural diagram of another file comparison device provided by an embodiment of the present application, including:

[0252] A division module 501 for dividing a first file and a second file to be compared into the same number of text blocks;

[0253] A sending module 502 for sending comparison data to multiple computing nodes. Among them, the comparison data sent to each computing node includes a first text block and a second text block. The block division order of the first text block in the first file is the same as the block division order of the second text block in the second file. The comparison data is used to trigger the computing node to mark the similarities and differences of the characters in the first text block and the second text block, and iteratively calculate the resource consumption situation when comparing the characters in the first text block and the second text block according to the similarity - difference marking result and the resource consumption iteration rule during character comparison; determine the block comparison result according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods during comparison. The block comparison result includes the character modification operations for updating the first text block into the second text block;

[0254] A receiving module 503 for receiving the block comparison results sent by the multiple computing nodes;

[0255] A generating module 504, configured to generate a file comparison result of the first file and the second file according to the comparison results of each block.

[0256] In some embodiments, the block comparison result further includes a block similarity. The generating module 504 is further configured to:

[0257] Determine a file similarity between the first file and the second file according to the block similarities sent by the multiple computing nodes and the number of characters of each second text block; and

[0258] The file similarity further includes the file similarity between the first file and the second file.

[0259] The division of modules in the embodiments of the present application is illustrative. It is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional module in the embodiments of the present application may be integrated in a processor, or may exist physically alone, or two or more modules may be integrated in one module. The coupling between each module can be realized through some interfaces, and these interfaces are usually electrical communication interfaces, but it does not exclude the possibility of being mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different positions of the same or different devices. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0260] After introducing the file comparison method and device of the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application will be introduced.

[0261] Next, refer to Figure 6 to describe the electronic device 130 implemented according to this embodiment of the present application. Figure 6 The displayed electronic device 130 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0262] As Figure 6 shown, the electronic device 130 is presented in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the above at least one processor 131, the above at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0263] The bus 133 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a processor, or a local bus using any bus structure in a multiple bus structure.

[0264] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0265] The memory 132 may also include a program / utilities 1325 having a set (at least one) of program modules 1324. Such program modules 1324 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0266] The electronic device 130 may also communicate with one or more external devices 134 (such as a keyboard, a pointing device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or may communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 135. Also, the electronic device 130 may communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 136. As shown in the figure, the network adapter 136 communicates with other modules for the electronic device 130 through a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0267] In an exemplary embodiment, the electronic device of the present application may at least include at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may be caused to execute the steps of any file comparison method provided in the embodiments of the present application.

[0268] In an exemplary embodiment, a storage medium is also provided. When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above file comparison methods. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.

[0269] In an exemplary embodiment, there is also provided a computer program product. When the computer program product is executed by an electronic device, the electronic device can implement any of the exemplary methods provided in this application.

[0270] It should be noted that although several modules or sub-modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more of the above-described modules can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0271] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0272] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0273] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0274] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also includes these changes and modifications.

Claims

1. A file comparison method, characterized in that: include: The computing node receives comparison data sent by the control node, the comparison data including a first text block and a second text block, the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file; marking the similarities and differences between the characters in the first text block and the second text block; Iteratively calculating resource consumption of the first text block and the second text block under different character comparison paths according to the similarities and differences marking results and the resource consumption iteration rule during character comparison; Determine a block comparison result according to the iterative calculation result and character comparison rules corresponding to different path backtracking modes, wherein the block comparison result includes a character modification operation performed when the first text block is updated to the second text block; The block comparison result is sent to the control node, triggering the control node to determine a file comparison result of the first file and the second file according to the block comparison results sent by the plurality of computing nodes.

2. The method according to claim 1, characterized in that Marking the similarities and differences between characters in the first text block and the second text block includes: For the i-th character ai in the first text block and the j-th character bj in the second text block: when ai and bj are the same, the difference and similarity marking value C[i][j] of ai and bj is determined to be a first preset value; when ai and bj are different, the difference and similarity marking value C[i][j] of ai and bj is determined to be a second preset value; Among them, 1≤i≤m, 1≤j≤n, m is the number of characters included in the first text block, and n is the number of characters included in the second text block.

3. The method according to claim 2, characterized in that According to the similarities and differences marking results and the resource consumption iteration rule during character comparison, the resource consumption of the first text block and the second text block under different character comparison paths is iteratively calculated, including: For ai and bj, the smallest value is selected from the accumulated values ​​of resource consumption of the first text block and the second text block under different character comparison paths as a reference accumulated value; According to the reference cumulative value and the difference and similarity marking values ​​C[i][j] of ai and bj, the cumulative value P[i][j] of resource consumption when comparing ai and bj is determined.

4. The method according to claim 3, characterized in that The first preset value is less than the second preset value, and the block comparison result is determined according to the iterative calculation result and the character comparison rules corresponding to the different path backtracking modes, including: Taking P[m][n] as a first cumulative value P[x][y], selecting a minimum value from the previous cumulative value of the first cumulative value as a second cumulative value, wherein the previous cumulative value includes at least one of P[x-1][y], P[x-1][y-1] and P[x][y-1]; Determine a character comparison result according to a path backtracking method from the first accumulated value to the second accumulated value and a character comparison rule corresponding to the path backtracking method; The second cumulative value is used as the new first cumulative value, and the step of selecting the smallest value from the previous cumulative value of the first cumulative value as the second cumulative value is performed until the first cumulative value has no previous cumulative value, and the results of each character comparison are combined to obtain the block comparison result.

5. The method according to claim 4, characterized in that Selecting the smallest cumulative value from the previous cumulative value of the first cumulative value as the second cumulative value includes: If there are at least two smallest accumulated values ​​in the previous accumulated values, and the at least two smallest accumulated values ​​include P[x-1][y-1], then P[x-1][y-1] is determined as the second accumulated value.

6. The method according to any one of claims 1 to 5, characterized in that: The comparison data also includes the context of the first text block and the context of the second text block, and before sending the block comparison result to the computing node, further includes: If it is determined that the block comparison result meets the re-comparison condition, the first text block is updated based on the context of the first text block, and the second text block is updated based on the context of the second text block; For the updated first text block and the second text block, the step of marking the similarities and differences of the characters in the first text block and the second text block is performed until the obtained block comparison result does not meet the re-comparison condition.

7. The method according to claim 6, characterized in that The re-matching condition includes that the block comparison result indicates that the character modification operation is distributed in the head area and / or the tail area of ​​the second text block.

8. A text comparison method, characterized in that: include: The control node divides the first file and the second file to be compared into the same number of text blocks; Sending comparison data to multiple computing nodes, wherein the comparison data sent to each computing node includes a first text block and a second text block, the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file, the comparison data is used to trigger the computing node to mark the similarities and differences of characters in the first text block and the second text block, and iteratively calculate the resource consumption when comparing the characters in the first text block and the second text block according to the similarities and differences marking results and the resource consumption iteration rule when comparing the characters; determining the block comparison result according to the iterative calculation result and the character comparison rule corresponding to different path backtracking methods when comparing, the block comparison result includes the character modification operation performed when the first text block is updated to the second text block; Receiving block comparison results sent by the multiple computing nodes; A file comparison result of the first file and the second file is generated according to the comparison results of each block.

9. The method according to claim 8, characterized in that The block comparison result also includes block similarity, and also includes: Determining the file similarity between the first file and the second file according to the similarities of the blocks sent by the plurality of computing nodes and the number of characters in each second text block; and The file similarity also includes the file similarity between the first file and the second file.

10. A file comparison device, applied to a computing node, characterized in that: include: A receiving module, configured to receive comparison data sent by a control node, wherein the comparison data includes a first text block and a second text block, wherein a block division order of the first text block in the first file is the same as a block division order of the second text block in the second file; A marking module, used for marking the similarities and differences between the characters in the first text block and the second text block; a consumption determination module, configured to iteratively calculate resource consumption when comparing characters in the first text block and the second text block according to the similarities and differences marking results and the resource consumption iteration rule when comparing characters; A comparison module, used for determining a block comparison result according to the iterative calculation result and character comparison rules corresponding to different path backtracking modes during comparison, wherein the block comparison result includes a character modification operation performed when the first text block is updated to the second text block; The sending module is used to send the block comparison result to the control node, and the control node determines the file comparison result of the first file and the second file according to the block comparison result.

11. A file comparison device, applied to a control node, characterized in that: include: A division module, used for dividing the first file and the second file to be compared into the same number of text blocks; A sending module, used for sending comparison data to multiple computing nodes, wherein the comparison data sent to each computing node includes a first text block and a second text block, the block division order of the first text block in the first file is the same as the block division order of the second text block in the second file, the comparison data is used to trigger the computing node to mark the similarities and differences of characters in the first text block and the second text block, and iteratively calculate the resource consumption when comparing the characters in the first text block and the second text block according to the similarities and differences marking results and the resource consumption iteration rules during the character comparison; determine the block comparison result according to the iterative calculation result and the character comparison rules corresponding to different path backtracking methods during the comparison, and the block comparison result includes the character modification operation performed when the first text block is updated to the second text block; A receiving module, used for receiving the block comparison results sent by the multiple computing nodes; A generating module is used to generate a file comparison result of the first file and the second file according to the comparison results of each block.

12. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 9.

13. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images