A document comparison method based on improved LCS algorithm

Abstract

The present application discloses a document comparison method based on an improved LCS algorithm, which relates to the field of data processing, including: receiving N documents to be compared uploaded by a user; dividing the N documents into N groups, each group comprising a main document and N-1 sub-documents; setting a minimum common subsequence length threshold L0; for each group, constructing a virtual two-dimensional array, with the number of characters in the main document as the number of rows, the number of characters in the sub-document as the number of columns, using sparse matrix storage, and only recording the matching character positions; starting from the elements in the last row and last column of the virtual two-dimensional array, traversing row by row to the elements in the first row and first column along the main diagonal direction of the two-dimensional array, recording the content of continuous matching characters, the starting position and the ending position in the main document and the sub-document; in view of the low efficiency of tobacco procurement document comparison in the prior art, the present application introduces optimization strategies such as document grouping, multi-threaded parallel comparison, cluster merging, etc., to improve the comparison efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a document comparison method based on an improved LCS algorithm. Background Art

[0002] With the deepening of economic globalization and increasingly fierce market competition, enterprises are more and more urgent to reduce procurement costs and improve procurement efficiency. As a common procurement method, bidding is widely used in engineering construction, material procurement, service outsourcing and other fields. Through bidding, the purchaser can choose suitable suppliers in a wider range and obtain high-quality products and services. For bidders, participating in bidding is an important way to open up the market and win orders.

[0003] However, in the actual bidding process, there are some urgent problems and challenges to be solved. Among them, checking for duplicate bids is a key link. Bids are the core documents for bidders to participate in bidding, representing the bidders' technical solutions, service commitments and quotation levels. In order to increase the probability of winning the bid, some bidders may directly copy the content of other bidders, resulting in a large number of duplicate bids. This behavior not only violates the principle of fair competition in bidding and disrupts the normal market order, but may also lead to the failure of the entire bidding process, causing losses to the purchaser and other bidders.

[0004] Taking the bidding of the tobacco industry as an example, due to the particularity and importance of tobacco products, the requirements for bid documents are very strict. The content of the bid document usually covers multiple links such as tobacco leaf planting, tobacco leaf procurement, cigarette production, logistics and distribution, and the technical parameters and process indicators must strictly comply with industry standards and corporate needs. If there is a large amount of repetitive content in the bid document, it will not only be impossible to accurately assess the actual capabilities of the bidder, but it may also lead to the cancellation of the bid, which will have an adverse impact on the production and operation of the enterprise.

[0005] At present, the technical means for checking for duplicate content in bid documents are relatively limited. Traditional manual review methods are time-consuming and labor-intensive, and are difficult to handle with massive bid documents. Common automatic duplicate checking tools based on keyword matching can only find completely identical text fragments, and lack the ability to identify synonym replacements, sentence structure adjustments, etc. In addition, existing duplicate checking systems can usually only provide simple indicators such as repetition rate, lack intuitive visual display, and are not conducive to users quickly locating and modifying duplicate content. Summary of the invention

[0006] In response to the problem of low efficiency in tobacco procurement document comparison in the prior art, the present application provides a document comparison method based on an improved LCS algorithm, introduces optimization strategies such as document grouping, multi-threaded parallel comparison, and clustering merging, and uses UUID as a request identifier to achieve real-time query of comparison progress and results, thereby improving comparison efficiency.

[0007] The purpose of this application is achieved through the following technical solutions.

[0008] The present application provides a document comparison method based on an improved LCS algorithm, comprising: S1, receiving N documents to be compared uploaded by a user, where the upload path is a local document path, a network URL or a document server; the document type is in Microsoft Word format; S2, dividing the N documents into N groups, each group comprising a main document and N-1 sub-documents; wherein the main document represents the first document of the corresponding group; and the sub-document represents the remaining documents of the corresponding group except the main document; S3, setting a minimum common subsequence length threshold L0 for judging repeated content; receiving stop words preset by the user; S4, for each group, constructing a virtual two-dimensional array, with the number of characters of the main document as the number of rows and the number of characters of the sub-document as the number of columns, using sparse matrix storage, and only recording the matching character positions; starting from the elements of the last row and last column of the virtual two-dimensional array, traversing row by row to the elements of the first row and first column along the main diagonal direction of the two-dimensional array, recording the content of the continuously matched characters, the starting position and the ending position in the main document and the sub-document; S5, performing multi-threaded parallel processing on the N groups generated by S2 , each thread calls step S4 to compare the main document and the secondary document in a group; the group comparison results obtained by each thread are passed to step S6; S6, for each group, the comparison results of each secondary document in the corresponding group with the main document are merged to generate a preliminary comparison result containing all repeated contents; S7, in the preliminary comparison result, the matching content of the duplicate report is identified and removed to obtain the final comparison result of the corresponding group; wherein the final comparison result includes the location of the duplicate content in the main document and the secondary document of the source; S8, in the main document of each group, the duplicate content in the final comparison result generated by step S7 is annotated using the revision mode and the highlighted background to generate the annotated result document; S9, a 32-bit UUID is generated as the request ID for the user to query the comparison progress and comparison results; the request ID is returned to the user, and the corresponding comparison progress and results are retrieved according to the request ID; S10, according to the comparison results of each group generated in step S7, a comparison report is generated, and the comparison report includes the basic data of the document, the overall duplication rate of the document, the duplication rate within the group, the distribution of duplicate content and the full text comparison.

[0009] Among them, LCS is the abbreviation of Longest Common Subsequence, that is, the longest common subsequence algorithm. It is a classic dynamic programming algorithm used to solve the longest common subsequence of two sequences. In this application, the character sequences of the two documents are regarded as two sequences to be matched. Through the LCS algorithm, we can find the longest common segment of the two documents at the character level, so as to detect and locate duplicate content. A virtual two-dimensional array refers to a logical two-dimensional matrix whose number of rows is equal to the number of characters in the primary document and the number of columns is equal to the number of characters in the secondary document. Each element in the matrix corresponds to a character position of the primary document and the secondary document. If the characters at these two positions are the same, they are marked as 1 in the matrix to indicate a match; otherwise, they are marked as 0. However, since the actual number of document characters may be large, it is very memory-consuming to directly open up a full-size two-dimensional array. Therefore, the system adopts a sparse matrix storage strategy, which only records the element positions with a value of 1 in the matrix, without storing a large number of 0s. The design of this virtual two-dimensional array vividly depicts the distribution of character matching between the two documents, and is an important basis for the subsequent extraction of continuous common substrings. At the same time, with the help of compressed storage of sparse matrices, the system's memory consumption is greatly reduced and processing efficiency is improved.

[0010] UUID is the abbreviation of Universally Unique Identifier. It is a string used to identify information in software systems and consists of 32 hexadecimal characters. The characteristic of UUID is global uniqueness, that is, UUIDs generated by any system at any time, any place, will not be repeated. It is often used for data identification and request tracking in distributed systems. In this application, after receiving the user's comparison request, the system immediately generates a 32-bit UUID string as the unique identifier of the request. This request ID will be returned to the user as is, and the user can use this ID to query the progress and results of the comparison task. At the same time, the system will also associate the request ID with the specific comparison task in the background and record it in the request context database. In this way, even if the comparison task is executed asynchronously, the system can accurately find the corresponding input document, intermediate results and final report based on the request ID.

[0011] Further, S2 divides the N documents into N groups, each group contains a main document and N-1 sub-documents, including: according to the number or submission time order of the N documents to be compared, the i-th document is used as the main document of the i-th group, where i is a positive integer; for each group, all documents except the main document are used as sub-documents of the corresponding group; according to the grouping result, a list groups consisting of N groups is created, each group contains a main document and N-1 sub-documents; traverse the N documents, take the i-th document as the main document, and create a new grouping object group; encapsulate the i-th document as the main document object mainDoc, and add it to group; traverse the N documents, when j is not equal to i, encapsulate the j-th document as the sub-document object subDocs, and add it to the sub-document object list of group; add the grouping object group to the grouping list groups; for each group group in the grouping list groups, extract the main document object mainDoc and the sub-document object subDocs list therein, and construct a dictionary structure groupdict containing the main document path and the sub-document path list; associate the dictionary structure groupdict with the request ID.

[0012] Among them, when N documents are grouped, the system regards each document as an independent comparison subject, called the main document. Other documents compared with the main document are called secondary documents. Specifically, in the i-th group, the i-th document is selected as the main document, and the other N-1 documents are secondary documents of the main document. The main document and the secondary document are both logical concepts and can actually be documents in any order. This division of main and secondary documents is to use the main document as a reference system in the subsequent comparison process to check the duplication of other documents with it one by one. Each document has the opportunity to become the main document and be checked by all other documents, thereby achieving a comprehensive pairwise comparison.

[0013] A grouping object is a data structure that contains a main document and sub-documents. Each grouping object corresponds to a specific comparison task, which determines which documents will be compared with the main document. In the implementation of this application, the grouping object is represented by a group structure, which contains two fields: mainDoc: represents the main document object of the group; subDocs: represents the list of sub-document objects of the group, which is an array of subDoc objects; through such a structured representation, each grouping object clearly describes a complete comparison context and clarifies the subject and object of the comparison. In the subsequent algorithm processing, the system can directly extract the main and sub-document information in the grouping object and execute the corresponding comparison logic.

[0014] A dictionary structure, also called a mapping or associative array, is a data structure used to store key-value pairs. It allows access to the corresponding value through a unique key, and supports fast insertion, deletion, and search operations. In the implementation of this application, the system uses a dictionary structure to organize and manage the document path information of each group. Specifically, for each group object group, the system extracts the main document path and the sub-document path list, and constructs a dictionary groupdict containing two key-value pairs: "main_doc": represents the path of the main document, and the corresponding value is a string; "sub_docs": represents the path list of the sub-document, and the corresponding value is a string array; through this dictionary structure, the system can use a unified data format to organize the document paths of different groups, which is convenient for subsequent storage, transmission, and parsing. At the same time, the dictionary structure also provides a convenient key-value access method, so that the system can query and extract the main and sub-document paths of a group at any time as needed.

[0015] Further, S3, setting the minimum common subsequence length threshold L0, including: counting the average paragraph length of each document to be compared; sorting the array arrAvgParaLen composed of the average paragraph lengths of all documents, and using the middle value of the sorted array arrAvgParaLen as the median medianLen: medianLen = arrAvgParaLen[N / 2]; setting the initial value k0 of the threshold coefficient, setting the initial value L00 of the minimum common subsequence length threshold L0: L00 = k0 *medianLen; the value range of k0 is 0.3 to 1.5. The minimum common subsequence length threshold L0 is set according to k0 and L00.

[0016] Furthermore, the minimum common subsequence length threshold L0 is set according to k0 and L00, through the following formula: L0 = k* medianLen, where k is the threshold coefficient; k = k0 + λ* simThresh; where λ is the adjustment coefficient, ranging from 0.1 to 0.5; simThresh is the threshold of the similarity score, ranging from 0.6 to 0.9.

[0017] Further, S4, for each group, construct a virtual two-dimensional array, and only record the matching character positions, including: for each group in the group list groups, obtain the main document mainDoc and the sub-document list subDocs therein; for the main document mainDoc and each sub-document subDoc in the sub-document list subDocs, construct a virtual two-dimensional array matrix: use the number of characters in the main document mainDoc as the number of rows m of the virtual two-dimensional array matrix; use the number of characters in the sub-document subDoc as the number of columns n of the virtual two-dimensional array matrix; use a sparse matrix to store the virtual two-dimensional array matrix, and only record the positions of the matching characters in the main document mainDoc and the sub-document subDoc; start from the element matrix[m-1][n-1] in the last row and last column of the virtual two-dimensional array matrix, and traverse along the main diagonal direction of the two-dimensional array matrix, row by row to the element matrix[0][0] in the first row and first column; record consecutive The content of the matching characters and the starting and ending positions of the corresponding content in the main document mainDoc and the sub-document subDoc are used to construct a matching result object match: the continuously matched character content is recorded as match.content; the starting position of the matching content in the main document mainDoc is recorded as match.main_start; the ending position of the matching content in the main document mainDoc is recorded as match.main_end; the starting position of the matching content in the sub-document subDoc is recorded as match.sub_start; the ending position of the matching content in the sub-document subDoc is recorded as match.sub_end; the matching result object match is added to the matching result list matchList as the matching result of the main document mainDoc and the current sub-document subDoc; after traversing all the sub-documents in the sub-document list subDocs, the matching result list matchList is used as the matching result of the current group group.

[0018] Specifically, this application introduces a virtual two-dimensional array data structure when performing a detailed comparison of each group, using the number of characters in the main document and the secondary document as the number of rows and columns, respectively, to vividly describe the positional relationship of character matching between the two documents. However, considering that the number of characters in an actual document may be large, it is too expensive to directly open up a full-size two-dimensional array storage space, so the system adopts a sparse matrix storage optimization strategy. The so-called sparse matrix storage refers to recording and storing only the position information of elements whose values ​​are not 0 in a two-dimensional array, without allocating storage space for these zero-value elements. This strategy is optimized for situations where the proportion of zero-value elements in the matrix is ​​high, and memory consumption is greatly reduced through compressed storage.

[0019] In the semantic vector matching scenario of this application, most of the positions in the virtual two-dimensional array are unmatched and are represented by 0, so it is very suitable to be stored in a sparse matrix. The system defines a SparseElement structure for each non-zero element (i.e., matching position) in the virtual two-dimensional array. The structure contains three fields: value: the value of the matrix element, which is always 1 in this scenario, indicating that the character at this position matches; row: the row number of the element, indicating the position of the matching character in the primary document; col: the column number of the element, indicating the position of the matching character in the secondary document; the system uses a vector <sparseelement>To store all non-zero elements in the virtual two-dimensional array, each element in the vector represents a matching position. In this way, most of the zero-value positions in the virtual two-dimensional array do not need to be stored, which greatly saves memory space. In the specific matching process, the system traverses the characters of the primary document and the secondary document and compares them one by one. Once a character match is found at a certain position, a SparseElement object is constructed, the row and column number of the position is recorded, and the object is added to the sparseMatrix vector. In this way, after traversing the two documents, the sparseMatrix vector stores all the matching position information.

[0020] In the subsequent continuous matching extraction stage, the system starts from the lower right corner of the virtual two-dimensional array (the last row and the last column) and traverses the sparseMatrix vector in reverse order along the main diagonal direction. Since the elements in the vector themselves contain row and column number information, the system can directly determine whether the current element is continuous with the previous element in position, and then extract the substring consisting of continuous matching characters. This compressed storage and traversal method based on sparse matrices avoids the operation of full-size two-dimensional arrays, greatly improving the space and time efficiency of the matching algorithm. From the perspective of implementation, this application makes full use of the basic characteristics of the C++ language such as vector containers and structures, encapsulates the position information of sparse matrix elements through a custom SparseElement structure, and then uses the vector container to realize the dynamic storage of non-zero elements, forming a set of concise and efficient sparse matrix storage solutions. This solution cleverly avoids the explicit opening and initialization of two-dimensional arrays, and only records the necessary matching positions. The time and space complexity are close to the linear scale of the number of document characters, and the advantages are particularly obvious when processing long documents. At the same time, when traversing the matching position, this application starts from the lower right corner of the matrix and searches in reverse order along the main diagonal direction. This traversal order can easily extract continuous matching strings. The system does not need to store and update string coordinates additionally, and all information can be directly obtained from the SparseElement object, further reducing the difficulty of implementing the matching logic.

[0021] Furthermore, for the main document mainDoc and each sub-document subDoc in the sub-document list subDocs, a virtual two-dimensional array matrix is ​​constructed, including: performing structured parsing on the main document mainDoc and the sub-document subDoc to obtain key data, which includes tobacco variety, quantity, unit price, origin and purchase time; normalizing the extracted key data; constructing a structured document object docObj according to the normalized key data; using the structured document object constructed by the main document mainDoc as the row label of the virtual two-dimensional array matrix; using the structured document object constructed by the sub-document subDoc as the column label of the virtual two-dimensional array matrix; comparing the corresponding fields of the main document structured object and the sub-document structured object, and calculating the similarity of each field; taking the weighted average of the calculated similarity values ​​of each field to obtain the overall similarity between the main document structured object and the sub-document structured object as the value of the corresponding position in the virtual two-dimensional array matrix.

[0022] Further, S6, for each group, merge the comparison results of each secondary document and the main document in the corresponding group to generate a preliminary comparison result containing all repeated contents, including: extracting all matching result objects in the group matching results to form a matching result object list matchList; calculating the similarity between each matching result object match in the matching result object list matchList and other matching result objects; constructing a similarity matrix simMatrix according to the similarity, and the row and column indexes correspond to the positions of the matching result objects in the matchList; clustering the matching result objects according to the similarity matrix simMatrix to obtain a clustering cluster list clusterList; for each cluster cluster in the clustering cluster list clusterList, extracting all matching result objects in the cluster cluster cluster, and Sort the matching lengths from large to small; select the matching result object with the largest matching length as the base object baseMatch; traverse the matching objects in the cluster cluster, and extract the difference between the matching object and the base object baseMatch: calculate the difference between the matching object and the base object in terms of content and position; extract the difference as an independent matching result object, and add it to the subsidiary object list of baseMatch; take the base object baseMatch and the corresponding subsidiary object list as the merged matching result objects, and add them to the merged matching result list mergedMatchList; after traversing all the clusters, sort the matching result objects in the merged matching result list mergedMatchList according to their starting positions in the main document, and obtain the preliminary comparison results prelimResults of the corresponding groups.

[0023] Furthermore, the matching result objects are clustered according to the similarity matrix simMatrix to obtain a clustering cluster list clusterList, including: taking each matching result object match in the matching result object list matchList as an initial clustering cluster cluster, each clustering cluster only contains one matching object; traversing the similarity matrix simMatrix to obtain the two clustering clusters clusterA and clusterB with the current maximum similarity simAB; calculating the average similarity meanSimA between the matching objects in the clustering cluster clusterA; calculating the average similarity meanSimB between the matching objects in the clustering cluster clusterB; if the similarity simAB is greater than a preset similarity threshold simThreshold, and is greater than meanSimA and meanSimB at the same time, merging clusters clusterA and clusterB to obtain a new cluster clusterM; repeating the merging steps until there is no clustering cluster pair that meets the merging condition in the similarity matrix simMatrix; and outputting the clustering cluster list clusterList as the result of clustering the matching result objects.

[0024] Further, the matching objects in the cluster cluster are traversed to extract the difference between the matching objects and the reference object baseMatch: the difference between the matching objects and the reference object in terms of content and position is calculated, including: initializing the subMatchList of the reference object baseMatch as an empty list; traversing each matching object match in the cluster cluster, recording the text content of the matching object match as matchText, and recording the text content of the reference object baseMatch as baseText; using the Needleman-Wunsch algorithm to extract the longest common subsequence LCS of matchText and baseText; taking the substring in matchText that is not included in LCS as the difference part diffText1; take the substring in baseText that is not included in LCS as the difference part diffText2; if diffText1 is located at the front of matchText, create a new match result object subMatch1, whose starting position is the starting position of match and whose ending position is the ending position of diffText1; if diffText1 is located at the back of matchText, create a new match result object subMatch2, whose starting position is the starting position of diffText1 and whose ending position is the ending position of match; add the extracted independent match result object subMatch to the subsidiary object list subMatchList of the reference object baseMatch.

[0025] Specifically, let the starting position of diffText1 in matchText be startPos1, and the ending position be endPos1; the starting position of matchText be startPosM. If the following conditions are met, diffText1 is said to be located at the front of matchText: startPos1 = startPosM; that is, the starting position of diffText1 is the same as the starting position of matchText. In other words, diffText1 is at the very front of matchText, and matchText starts with diffText1. Let the starting position of diffText1 in matchText be startPos1, and the ending position be endPos1; the ending position of matchText is endPosM. If the following conditions are met, diffText1 is said to be located at the end of matchText: endPos1 = endPosM; that is, the ending position of diffText1 is the same as the ending position of matchText. In other words, diffText1 is at the very back of matchText, and matchText ends with diffText1.

[0026] Further, S9 generates a 32-bit UUID as a request ID for users to query the comparison progress and comparison results, including: using a UUID algorithm based on timestamp, hardware identification code and random number to generate a 32-bit request ID string as a unique identifier of the set of documents to be compared; associating the generated request ID with the storage path or URL of the N documents to be compared received in step S1, building a request context structure, and storing it in a request context database; returning the generated request ID string to the user who submitted the comparison request through the network interface as the unique credential for the user to subsequently retrieve the comparison task progress and the final output results.

[0027] Compared with the prior art, the advantages of this application are:

[0028] By grouping the documents to be compared, each group contains a main document and multiple sub-documents, which avoids repeated comparisons and reduces unnecessary calculations. At the same time, multi-threaded parallel processing of group comparison tasks makes full use of computing resources and further improves the comparison speed.

[0029] Adopting an adaptive threshold strategy, the minimum common subsequence length threshold is dynamically adjusted according to the median of the average paragraph length of the document. This strategy can automatically adapt to documents of different sizes and styles, and can more accurately identify duplicate content for document collections with large differences in paragraph length.

[0030] The improved LCS algorithm uses a virtual two-dimensional array to store matching positions and only records matching characters, avoiding the need to store a complete comparison matrix and significantly reducing memory consumption. For long document comparisons, the storage space required by the traditional LCS algorithm grows quadratically with the length of the document, while the space complexity of the improved LCS algorithm is only linearly related to the number of matching characters.

[0031] After the group comparison is completed, a clustering algorithm based on similarity is used to merge the matching results within the group and aggregate highly similar matching content. Then, through difference extraction, the difference between the clustered matching content and the benchmark object is identified and marked separately as an auxiliary matching item. This process can effectively eliminate duplicate reports and redundant content in the comparison results, making the results more refined and accurate.

[0032] UUID is introduced as the request identifier, and the generated unique request ID is associated with the document to be compared and stored in the request context database. Users can query the progress and status of the comparison task in real time through the request ID to understand the comparison completion percentage. This method makes the document comparison process transparent and convenient for users to track and manage, especially in time-consuming scenarios such as batch document comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0034] Figure 1 is an exemplary flow chart of a document comparison method based on an improved LCS algorithm according to some embodiments of the present application;

[0035] Figure 2 It is a data processing schematic diagram of a document comparison method based on an improved LCS algorithm according to some embodiments of the present application;

[0036] Figure 3 This is an exemplary flowchart of setting a minimum common subsequence length threshold according to some embodiments of the present application. DETAILED DESCRIPTION

[0037] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0038] like Figure 1 and Figure 2 As shown, a document comparison method based on an improved LCS algorithm receives N documents to be compared uploaded by a user, where the upload path is a local document path, a network URL or a document server; the document type is in Microsoft Word format; the N documents are divided into N groups, each group containing a main document and N-1 sub-documents; wherein the main document represents the first document of the corresponding group; the sub-document represents the remaining documents of the corresponding group except the main document; a minimum common subsequence length threshold L0 is set to judge duplicate content; stop words preset by the user are received; for each group, a virtual two-dimensional array is constructed, with the number of characters in the main document as the number of rows and the number of characters in the sub-document as the number of columns, sparse matrix storage is used, and only the matching character positions are recorded; starting from the element in the last row and last column of the virtual two-dimensional array, along the main diagonal direction of the two-dimensional array, row by row traverses to the element in the first row and first column, and records the content of continuous matching characters, the starting position and the ending position in the main document and the sub-document; multi-threaded parallel processing is performed on the generated N groups, and each thread calls the virtual two-dimensional array to compare the main document and the sub-document in a group. The method comprises the following steps: first, comparing the primary and secondary documents of the corresponding group with the secondary documents of the corresponding group; second, comparing the secondary documents of the corresponding group with the primary document; for each group, merging the comparison results of the secondary documents in the corresponding group with the primary document to generate a preliminary comparison result containing all duplicate contents; identifying and removing the matching contents of the duplicate reports in the preliminary comparison result to obtain the final comparison result of the corresponding group; wherein the final comparison result includes the location of the duplicate contents in the primary document and the secondary documents of the source; in the primary document of each group, using the revision mode and the highlighted background to mark the duplicate contents in the generated final comparison result, and generating a marked result document; generating a 32-bit UUID as a request ID for users to query the comparison progress and comparison results; returning the request ID to the user, and retrieving the corresponding comparison progress and results according to the request ID; generating a comparison report according to the generated comparison results of each group, and the comparison report includes the basic data of the document, the overall duplication rate of the document, the duplication rate within the group, the distribution of duplicate contents and the full-text comparison.

[0039] Specifically, the system receives N documents to be compared uploaded by the user. The user can choose to upload the document file from the local hard disk, or provide the network URL address of the document or the access path of the document server. The system supports the Microsoft Word format, including the two common suffixes .doc and .docx. Local upload: The user clicks the "Select File" button to open the local file selection dialog box, browse and select the Word document to be uploaded. It supports the simultaneous selection of multiple files, or the accumulation of documents to be compared through multiple selections. URL upload: The page provides a URL input box, in which the user can paste or enter the network address of the Word document. The system supports HTTP and HTTPS protocols. After the user enters the URL and clicks the "Upload" button, the system automatically downloads the Word document corresponding to the URL. At the same time, considering the needs of batch upload, URL upload also supports batch mode. The user can enter multiple URL addresses in the text box, separated by line breaks. Document server: For users with existing document management systems, the upload interface provides configuration items for the document server. The user can enter the IP address, port number, access path and other information of the document server. The system uses common protocols such as WebDAV to directly obtain the Word file specified by the user from the document server.

[0040] S2, after receiving N Word documents uploaded by the user to be compared, the system automatically groups these N documents. The purpose of grouping is to reduce unnecessary repeated comparisons and improve the overall comparison efficiency. First, according to the upload order or number of the documents, each document is regarded as a main document, and a comparison group with it as the core is constructed. Taking the numbering as an example, the i-th uploaded document is used as the main document of the i-th group, where i is a positive integer from 1 to N. The secondary documents of this group are all documents except the i-th document. In this way, N documents constitute N comparison groups, each of which contains a main document and N-1 secondary documents.

[0041] In terms of implementation, the grouping module defines two data structures: Group and Document. Group represents a comparison group, which contains a main document attribute mainDoc and a sub-document list subDocs. Document represents a document, which contains metadata attributes such as the document path, number, and upload time. The core of the grouping algorithm is two nested loops. The outer loop traverses N documents, takes each document as the main document, and creates a new Group object. The inner loop traverses N documents again, takes all documents except the main document as sub-documents, encapsulates them into Document objects, and adds them to the Group's subDocs list. Through the above nested loops, N documents are divided into N Group objects and stored in a list called groups. Each Group object contains a main document mainDoc and a sub-document list subDocs, which fully reflects the structural characteristics of grouping.

[0042] To facilitate subsequent document comparison operations, after generating the Group object list groups, the grouping module further converts it into a data format that is more suitable for transmission and storage. Specifically, it traverses each Group object in the groups list, extracts the main document object mainDoc and the sub-document object list subDocs, and constructs a dictionary structure groupdict containing the main document path and the sub-document path list. In the groupdict dictionary, the key is main, and the value is the file path of the main document mainDoc object; the key is sub, and the value is a list consisting of the file paths of each sub-document object in the sub-document subDocs list. This dictionary structure clearly distinguishes the different roles of the main document and the sub-document through the key, and the value directly gives the corresponding file path, avoiding the need to extract path information in subsequent operations.

[0043] Since Python's built-in dictionary type dict uses an efficient hash storage and search algorithm, the groupdict dictionary can quickly access and extract the primary and secondary document paths. At the same time, Python dictionaries can also be easily converted to JSON format, which is convenient for transmission and storage between different modules. While generating the groupdict dictionary, the grouping module also generates a unique request ID for each upload and comparison task. The request ID is generated using the UUID (Universally Unique Identifier) ​​algorithm to ensure global uniqueness and no duplication. UUID is a 128-bit integer, usually represented as 32 hexadecimal digits separated by 5 hyphens, such as: abc12345-67cd-90ef-0123-456789abcdef. The grouping module uses the Python client library redis-py provided by Redis to establish a connection with the Redis server. Through the set method provided by redis-py, the request ID is used as the key and the groupdict dictionary as the value, and stored in Redis in the form of a string. When a query is needed, the corresponding groupdict dictionary is obtained by the get method according to the request ID, and the json.loads method is used to convert it into a dictionary object in Python. The grouping module constructs the request ID and the corresponding groupdict dictionary as a key-value pair, so that the grouping information of the task can be quickly queried and located according to the request ID. Considering the high frequency of reading and writing of grouped data, the grouping module chooses to store it in NoSQL databases such as Redis. Compared with traditional relational databases, NoSQL databases use flexible data models such as key-value pairs and column families.

[0044] like Figure 3 As shown in S3, in order to adaptively determine the minimum common subsequence length threshold L0, the system first needs to count the average paragraph length of each document to be compared. This is because the length of a paragraph, as a natural segmentation unit of a document, can better reflect the content characteristics of the document, such as the amount of information, the complexity of the language, etc. Specifically, the system uses segmentation algorithms in natural language processing, such as paragraph indentation-based, semantic similarity-based, etc., to parse each document into a paragraph list. Each paragraph is an independent string containing several sentences or semantically complete language blocks. Then, the system traverses the paragraph list and counts the character length of each paragraph. The character length here refers to the total number of characters in the paragraph string, including all characters such as letters, numbers, punctuation marks, and spaces. The system accumulates the paragraph lengths and divides them by the total number of paragraphs to obtain the average paragraph length of the document. For example, a document contains 5 paragraphs, whose character lengths are 100, 150, 120, 180, and 90 respectively. The average paragraph length of the document is: (100+150+120+180+90) / 5=128. The system repeats the above process for all documents to be compared, thus obtaining a set of average paragraph lengths. In order to further obtain a global reference index, the system sorts this set in order from small to large according to the average paragraph length, forming an ordered array arrAvgParaLen. There are 5 documents, whose average paragraph lengths are 100, 120, 150, 180, and 200 respectively. The sorted array arrAvgParaLen is [100, 120, 150, 180, 200]. For this ordered array, the system selects the element in the middle position as the median medianLen, that is: medianLen=arrAvgParaLen[N / 2]; where N is the total number of documents. When N is an odd number, the above formula is the middle element of the array; when N is an even number, the above formula takes the N / 2th element after sorting, which is slightly smaller than the true median, but has little effect on subsequent calculations. In the above example, medianLen=arrAvgParaLen[2]=150. The median is chosen as the global reference because it can effectively reduce the impact of extreme values, thereby reflecting the overall level more robustly. For example, in the above example, the average paragraph length of the fifth document is significantly larger. If the arithmetic mean (150) is used, L0 may be too high, resulting in failure to detect more repetitions; while the median (150) is not affected by the extreme value.

[0045] After obtaining the median medianLen, the system initializes the threshold coefficient k0 and calculates the initial value L00 of the minimum common subsequence length threshold L0 based on it. The calculation formula of L00 is: L00=k0*medianLen; where k0 is a preset constant that can be adjusted based on experience or experiments. In order to further optimize the threshold L0, the system also introduces the similarity score threshold simThresh to adjust k0. The adjusted threshold coefficient k is calculated according to the following formula: k=k0+λ*simThresh; where λ is the adjustment coefficient, ranging from 0.1 to 0.5; simThresh is the similarity score threshold, ranging from 0.6 to 0.9. Intuitively, when we expect the similarity of the documents to be higher (i.e., simThresh is larger), we should appropriately increase the threshold coefficient k, thereby increasing the minimum common subsequence length threshold L0. This can avoid judging some shorter common segments as duplicate content. Conversely, when we have lower requirements for similarity, we should appropriately reduce k and L0 to detect more duplicate content.

[0046] In addition to adaptively calculating the minimum common subsequence length threshold L0, the system also supports user-defined stop words to further improve the accuracy of duplicate judgment. This is particularly important for the comparison of bidding documents in tobacco industry procurement. In tobacco industry procurement, bidding documents often contain a large number of common words, such as "tobacco", "procurement", "bidding", "tender", "project", "company", etc. These words appear very frequently in different bidding documents, but they are of little significance for judging whether the document content is repeated. On the contrary, they may interfere with the accuracy of duplicate judgment. For example, if two bidding documents happen to have more continuous matches on these common words, but are not the same in other substantive content, the system may mistakenly judge them as duplicates. To solve this problem, the system allows users to define stop word lists. Users can add high-frequency words in tobacco industry procurement to the stop word list based on their business experience. These stop words can be single words or phrases. For example, users can define the following stop word list: ["tobacco", "procurement", "bidding", "tender", "project", "company", "limited company", "joint stock company", "ten thousand yuan", "yuan", "date", "for reference only"...]. When extracting the minimum common subsequence, the system automatically skips the words in the stop word list. Specifically, the system first performs word segmentation and part-of-speech tagging on the document to obtain a word list. Then, the system traverses the list and determines whether each word in it is in the stop word list. If it is, the word is directly skipped and not included in the calculation of the minimum common subsequence; if not, it is included in the subsequent calculation as a candidate word. If the stop word list is not used, the system may think that the two sentences are highly repeated because they have continuous matches on words such as "tobacco", "purchase", "Limited Company", "tobacco silk bundle", "purchase", "project", "open bidding", "bidder", and "pre-submitted bidding documents". But in fact, the two sentences are not repeated in substantive content because they describe bidding projects of different companies at different times. After introducing the above stop word list, the system will automatically filter out interference words such as "tobacco", "purchase", and "Limited Company", and find that the matching degree of the two sentences on the remaining words is not high, so it will not be judged as repeated.

[0047] S4, for each group, the system compares the main document and the sub-document by constructing a virtual two-dimensional array, and then obtains the matching results between them. This method makes full use of the structural characteristics of the procurement documents of the tobacco industry, and can quickly and accurately determine the duplication of documents. Specifically, for each group in the group list groups, the system first extracts the main document mainDoc and the sub-document list subDocs. Then, the system traverses the subDocs list and compares each sub-document subDoc with the main document mainDoc in turn.

[0048] When comparing the main document mainDoc and a sub-document subDoc, the system constructs a virtual two-dimensional array matrix. The number of rows m of the array is equal to the number of characters in the main document mainDoc, and the number of columns n is equal to the number of characters in the sub-document subDoc. However, considering that actual documents are often very large, directly opening a two-dimensional array with m rows and n columns will result in huge memory consumption. Therefore, the system adopts a sparse matrix storage method, that is, only recording the position and value of the elements in the array whose values ​​are not 0, without explicitly storing all elements.

[0049] In order to construct a virtual two-dimensional array matrix, the system needs to perform structural analysis on the main document mainDoc and the sub-document subDoc. Since procurement documents in the tobacco industry usually have relatively fixed formats and fields, some key data can be predefined, such as tobacco varieties, quantities, unit prices, origins, procurement time, etc. The system extracts these key data from the main document and sub-documents through rule or template matching. After extracting the key data, the system normalizes it to eliminate format differences between different documents. For example, for the quantity and unit price fields, the system converts them into a unified numeric type and unit; for the procurement time field, the system converts it into a unified date format.

[0050] Then, the system constructs structured document objects docObj for the main document mainDoc and the sub-document subDoc based on the normalized key data. The document object docObj contains multiple attributes, corresponding to the key data fields, such as docObj.variety for tobacco varieties, docObj.quantity for normalized quantity, docObj.unitPrice for normalized unit price, docObj.origin for origin, and docObj.purchaseTime for normalized purchase time.

[0051] In the virtual two-dimensional array matrix, the system uses the main document object docObj_main as the row label and the secondary document object docObj_sub as the column label. Then, the system compares the corresponding fields of docObj_main and docObj_sub and calculates the similarity between them as the value at the corresponding position in the virtual two-dimensional array matrix. The calculation of similarity varies according to the field type: the following comparison methods are adopted for the tobacco variety field variety, the quantity field quantity, and the unit price field unitPrice. For the tobacco variety field variety, the system adopts a strict string matching method. Specifically, the system first performs string preprocessing on the variety field values of the two documents, including removing leading and trailing spaces, converting to lowercase letters, removing special characters, etc., to eliminate format differences. Then, the system uses the equals method to compare whether the two preprocessed strings are exactly the same. If they are the same, the similarity value of the variety field is recorded as 1; otherwise, the similarity value is recorded as 0. This comparison method is simple and direct and can accurately identify exactly the same tobacco varieties. For example, if the variety field values of two documents are "Hongtashan" and "Hongtashan" respectively, after preprocessing, they are both converted to "hongtashan", and through the equals comparison, the system can conclude that they are exactly the same, so the similarity value is 1. However, if the variety value of one document is "Hongtashan" and the other is "Hongtashan 1", although they differ by only one character, through the equals comparison, the system will still consider them inconsistent, so the similarity value is 0.

[0052] For the quantity field quantity and the unit price field unitPrice, the system adopts a comparison method based on the relative difference percentage. First, the system normalizes the quantity values and unit price values of the two documents and converts them to a unified numerical unit. For example, quantity is converted to "tons" and unitPrice is converted to "yuan / ton". Then, the system calculates the relative difference percentage between the quantity values of the two documents, and the formula is as follows: ; where quantity1 and quantity2 respectively represent the normalized quantity values of the two documents, and max represents the maximum value function. The value range of quantityDiffPercentage is [0, 1], which represents the relative difference degree between the two quantity values. The greater the difference, the closer quantityDiffPercentage is to 1.

[0053] Similarly, the system calculates the relative difference percentage between the unit price values of the two documents, and the formula is as follows: ; where unitPrice1 and unitPrice2 represent the normalized unitPrice values ​​of the two documents respectively. Next, the system maps quantityDiffPercentage and unitPriceDiffPercentage to similarity values ​​between 0 and 1, and the mapping function is: similarityValue=1-diffPercentage; where diffPercentage represents quantityDiffPercentage or unitPriceDiffPercentage. This mapping function can convert the difference percentage into a similarity value. The larger the difference percentage, the closer the similarity value is to 0; the smaller the difference percentage, the closer the similarity value is to 1. For example, if the normalized quantity values ​​of two documents are 100 tons and 120 tons respectively, their relative difference percentage is: quantityDiffPercentage = abs(100-120) / max(100, 120) = 0.167; mapping it to the similarity value, we get: quantitySimilarity = 1-0.167 = 0.833; this shows that although the quantity values ​​of the two documents are not exactly the same, the difference between them is not large, so the similarity value is high, 0.833. Similarly, if the normalized unitPrice values ​​of the two documents are 800 yuan / ton and 1000 yuan / ton respectively, their relative difference percentage is: unitPriceDiffPercentage = abs(800-1000) / max(800, 1000) = 0.2; mapped to the similarity value, we get: unitPriceSimilarity = 1-0.2 = 0.8; this shows that although the unit values ​​of the two documents differ by 200 yuan / ton, this difference is not too large relative to their value ranges, so the similarity value is still relatively high, at 0.8.

[0054] The following comparison method is used for the origin field origin and the purchase time field purchaseTime. For the origin field origin, the system uses the edit distance algorithm to measure the similarity between two strings. Edit distance, also known as Levenshtein distance, refers to the minimum number of editing operations required to convert one string into another. The editing operations here include insertion, deletion, and replacement. The smaller the edit distance, the more similar the two strings are. Specifically, the system first performs string preprocessing on the origin field values ​​of the two documents, including removing leading and trailing spaces, converting to lowercase letters, removing special characters, etc., to eliminate format differences. Then, the system uses a dynamic programming algorithm to calculate the edit distance between the two preprocessed strings. Let the two strings be str1 and str2, and their lengths are len1 and len2 respectively. The system creates a two-dimensional matrix dp, where dp[i][j] represents the edit distance between the first i characters of str1 and the first j characters of str2. The initialization rules of the matrix dp are as follows: when i=0, dp[0][j]=j, which means that the empty string is converted to the first j characters of str2, and j insertion operations are required; when j=0, dp[i][0]=i, which means that the first i characters of str1 are converted to the empty string, and i deletion operations are required. After initialization, the system fills the matrix dp row by row according to the following state transition equation: if str1[i]=str2[j], then dp[i][j]=dp[i-1][j-1], indicating that the two characters match and no editing operation is required; if str1[i]≠str2[j], then dp[i][j]=min(dp[i-1][j], dp[i][j-1], dp[i-1][j-1])+1, indicating that the two characters do not match and need to be edited. The minimum value of the three operations of inserting (dp[i][j-1]), deleting (dp[i-1][j]), and replacing (dp[i-1][j-1]) can be selected, and then 1 is added. Finally, the lower right corner element dp[len1][len2] of the matrix dp is the edit distance between str1 and str2.

[0055] To convert the edit distance into a similarity value, the system defines a mapping function: originSimilarity = 1 - editDistance / max(len1, len2); where editDistance represents the edit distance between str1 and str2, and max(len1, len2) represents the maximum length of the two strings. This mapping function can normalize the edit distance to the range of 0 to 1 to obtain the similarity value of the origin field. The smaller the edit distance, the closer the similarity value is to 1; the larger the edit distance, the closer the similarity value is to 0. For example, if the origin field values of two documents are "a certain city" and "a certain", after preprocessing, they are both converted to "shanghai" and "shanghai". Through the edit distance algorithm, the system calculates that their edit distance is 0, which means that they can be fully matched without any editing operations. Mapping it to the similarity value, we get: originSimilarity = 1 - 0 / max(8, 8) = 1; this indicates that the origins of the two documents are exactly the same, and the similarity value reaches the maximum value of 1. Another example, if the origin field values of two documents are "a certain city" and "ten thousand yuan", after preprocessing, they are converted to "shanghai" and "jiangsu". Through the edit distance algorithm, the system calculates that their edit distance is 6, which means that 6 editing operations (replacements) are required to fully match. Mapping it to the similarity value, we get: originSimilarity = 1 - 6 / max(8, 7) = 0.25; this indicates that the origins of the two documents are quite different and require more editing operations to match, so the similarity value is relatively low, only 0.25.

[0056] For the purchase time field purchaseTime, the system uses a comparison method based on the difference in days. First, the system performs date parsing on the purchaseTime field values ​​of the two documents and converts them into a standard date format, such as ""yyyy-MM-dd"". Then, the system calculates the difference in days between the two dates using the following formula: daysDiff = abs(date1-date2); where date1 and date2 represent the parsed date values ​​of the two documents, respectively, and abs represents the absolute value function. The value range of daysDiff is [0, +∞), which represents the difference in days between the two dates. Next, the system maps daysDiff to a similarity value from 0 to 1, and the mapping function is: purchaseTimeSimilarity = 1 / (1+daysDiff); this is a nonlinear mapping function, and as daysDiff increases, purchaseTimeSimilarity will get closer and closer to 0, but will never be completely equal to 0. The characteristic of this function is that when daysDiff is small, purchaseTimeSimilarity decreases faster; when daysDiff is large, purchaseTimeSimilarity decreases slower. This is in line with the actual situation, because the difference in purchase time of a few days and a difference in months have different effects on document duplication. For example, if the purchaseTime field values ​​of two documents are "2022-01-01" and "2022-01-05" respectively, after date parsing, the system calculates their difference in days as: daysDiff = abs (2022-01-01-2022-01-05) = 4; mapping it to the similarity value, we get: purchaseTimeSimilarity = 1 / (1+4) = 0.2; this shows that the purchase time of the two documents differs by 4 days, and the similarity value is 0.2, indicating that their purchase times are relatively close, but not exactly the same. For example, if the purchaseTime field values ​​of two documents are "2022-01-01" and "2022-06-01" respectively, after date parsing, the system calculates their difference in days as: daysDiff = abs(2022-01-01-2022-06-01) = 151; mapping it to the similarity value, we get: purchaseTimeSimilarity = 1 / (1+151) = 0.0066; this shows that the purchase time of the two documents differs by 151 days, or about 5 months, and the similarity value is very close to 0, indicating that their purchase time difference is very large, and they can basically be considered to be completely different purchase behaviors.

[0057] After calculating the similarity of each field between the main document object and each secondary document object, the system uses a weighted average method to obtain the overall similarity between them and uses it as the value of the corresponding position in the matching matrix. Then, the system uses a dynamic programming algorithm, starting from the lower right corner of the matrix, traversing along the main diagonal direction to the upper left corner, extracting matching text fragments, and generating the final matching result. Specifically, for the main document object docObj_main and a secondary document object docObj_sub, the system calculates their similarity values ​​on key fields such as tobacco variety, quantity, unit price, origin, and purchase time, which are recorded as Then, the system assigns weight coefficients to these fields according to their business importance, which are recorded as , and satisfy: ; Next, the system calculates and The overall similarity of similarity is as follows: ; Through weighted averaging, the system can comprehensively consider the matching degree of each field and obtain an overall similarity value between 0 and 1. The closer the similarity value is to 1, the more similar the two document objects are; the closer the similarity value is to 0, the less similar the two document objects are. The system will and The overall similarity value similarity is assigned to the elements at the corresponding positions in the virtual two-dimensional array matrix, that is, matrix[i][j]=similarity, where i and j are and The subscript in the document object list. In this way, by traversing all the secondary document objects, the system constructs a similarity matrix between the main document object and all the secondary document objects.

[0058] After constructing the similarity matrix, the system uses a dynamic programming algorithm, starting from the lower right element of the matrix matrix[m-1][n-1], and traversing along the main diagonal direction to the upper left element matrix[0][0] to extract the matching text fragment. Specifically, the system uses two pointers i and j, which initially point to the last row and last column of the matrix, that is, i=m-1, j=n-1. Then, the system repeatedly performs the following operations: If the value of matrix[i][j] is greater than 0, it means that the i-th character of the main document mainDoc matches the j-th character of the sub-document subDoc, and is part of a continuous matching fragment. The system records the position of this matching character, then decrements i and j by 1 respectively, and continues to check whether the previous character matches. If the value of matrix[i][j] is equal to 0, it means that the character at the current position does not match, and a continuous matching fragment ends. The system determines whether the number of recorded matching characters is greater than or equal to the minimum matching length (such as 10 characters). If so, it takes it as a valid matching result and constructs a matching result object match: match.content is the matching text content, which corresponds to the [i+1, lastMatchedPos] substring of mainDoc and the [j+1, lastMatchedPos] substring of subDoc; match.main_start is the starting position of the matching text in mainDoc, i.e., i+1; match.main_end is the ending position of the matching text in mainDoc, i.e., lastMatchedPos; match.sub_start is the starting position of the matching text in subDoc, i.e., j+1; match.sub_end is the ending position of the matching text in subDoc, i.e., lastMatchedPos. The system adds the matching result object match to the matching result list matchList. Then, the system clears the recorded matching character positions, decrements i and j by 1 respectively, and starts looking for the next continuous matching fragment. Repeat until i or j is less than 0, that is, the entire similarity matrix is ​​traversed. After traversing the matrix, the system obtains all matching results between the main document mainDoc and the current sub-document subDoc, and stores them in matchList. The system adds matchList as the comparison result of the current document pair (mainDoc, subDoc) to the final comparison result set. Next, the system takes the next sub-document from the sub-document list subDocs and repeats the above steps until all sub-documents are traversed to obtain the comparison result between the main document mainDoc and each sub-document.

[0059] S5, the system adopts a multi-threaded parallel processing method to compare and analyze the N groups generated in step S2 to improve the processing efficiency of the entire system. Specifically, the system first creates a thread pool. The number of threads in the thread pool can be adjusted according to the hardware configuration and data scale of the system, for example, set to twice the number of CPU cores. Then, the system creates a comparison task for each group and submits the comparison task to the thread pool for processing. The thread pool assigns the comparison task to the idle worker thread according to the preset scheduling strategy (such as FIFO, priority, etc.). Each worker thread executes a comparison task, that is, compares and analyzes the procurement documents in a group. For the specific comparison process, see step S4, which will not be repeated here. The worker thread obtains the comparison result of the group by calling the comparison function defined in step S4, passing in parameters such as the group number, the main document object and the list of secondary document objects. The comparison result includes two parts: a matching result list matchList, each element in the list indicates that a text segment in the main document matches the corresponding segment of a secondary document, including the specific content of the match, the start and end positions in the main document and the secondary document, and other information. A similarity matrix simMatrix, the rows of the matrix represent the main document objects, the columns represent the secondary document objects, and the element values ​​represent the overall similarity between the corresponding document objects, with a value range of [0, 1]. The worker thread takes the matched matchList and simMatrix as the comparison result and encapsulates them into a comparison result object compResult. The worker thread adds the comparison result object compResult to the thread-safe blocking queue resultQueue as the return value of the comparison task. The blocking queue resultQueue can realize synchronization and communication between threads, ensuring that each comparison result can be safely received and processed by the main thread.

[0060] The main thread loop takes out the comparison result object from the resultQueue and passes it to step S6 for subsequent processing, such as generating a comparison report, updating the database, etc. At the same time, the main thread records the number of comparison tasks that have been processed. When all the comparison tasks are completed, the main thread sends a shutdown signal to the thread pool to recycle system resources. Through multi-threaded parallel processing, the system can make full use of the computing power of multi-core CPUs and significantly improve the speed and throughput of comparison analysis. At the same time, this application uses concurrent tools such as thread pools and blocking queues, which can effectively avoid the overhead of thread creation and destruction, and realize coordination and synchronization between threads, ensuring the stability and reliability of the system.

[0061] S6, after the system compares and analyzes the procurement documents in each group in step S5, it is necessary to merge and organize the comparison results of the corresponding group in step S6 to generate a preliminary comparison result containing all repeated contents. Specifically, for a certain group, the system first extracts the matching result objects from all the comparison results of the group to form a matching result object list matchList. Each element in matchList indicates that a text segment in the main document matches the corresponding text segment in a secondary document. Next, the system calculates the similarity between each pair of matching result objects in matchList. The calculation of similarity can comprehensively consider the consistency of the two matching objects in terms of text content, start and end positions, etc., for example, using commonly used similarity measurement methods such as edit distance and Jaccard coefficient. According to the calculated similarity values, the system constructs a two-dimensional similarity matrix simMatrix, the rows and columns of the matrix correspond to the matching result objects in matchList, and the element values ​​are the similarities between the corresponding two matching objects, with a value range of [0, 1]. imMatrix is ​​a symmetric matrix, and the element values ​​on the main diagonal are all 1, indicating that each matching object is completely similar to itself.

[0062] The system uses the constructed similarity matrix simMatrix to cluster the matching result objects. The purpose of clustering is to classify highly similar matching objects into the same cluster, forming a set of matching objects that are internally consistent and externally separated. The clustering results are presented in the form of a cluster list clusterList, and each cluster cluster in the list contains a group of matching fragments with high similarity.

[0063] The specific clustering steps are as follows: Initialization of the cluster list clusterList. At the beginning of clustering, the system regards each matching object match in the matching result object list matchList as an independent cluster cluster. At this time, each cluster cluster is a collection that contains only a single matching object. This is equivalent to setting a starting point for the clustering process, and subsequent clusters will be gradually merged and optimized on this basis. Traverse the similarity matrix simMatrix to find the two cluster clusters clusterA and clusterB with the largest current similarity. The element value at the intersection of the clusterA row and the clusterB column in the matrix is ​​the similarity between the two cluster clusters, recorded as simAB. By traversing the matrix, the system can find the pair with the highest similarity among all cluster cluster pairs as the preferred object for merging. Calculate the average similarity between the matching objects in cluster clusterA and clusterB respectively, recorded as meanSimA and meanSimB. The average similarity reflects the degree of consistency within the cluster cluster, which can be measured by the arithmetic mean of the similarity of all pairwise matching objects in the cluster cluster. The higher the average similarity, the more similar the matching objects in the cluster are, and the better the clustering effect. By calculating the internal average similarity of clusterA and clusterB, the system can evaluate their respective cohesion. Determine whether clusterA and clusterB meet the merging conditions. The merging conditions include two criteria: first, the similarity simAB of clusterA and clusterB must be greater than the preset similarity threshold simThreshold; second, simAB must be greater than the internal average similarities meanSimA and meanSimB of clusterA and clusterB at the same time. The first criterion ensures that only clusters that are similar enough can be merged, avoiding the forced bundling of unrelated matching objects; the second criterion ensures that the internal consistency of the merged cluster is still high, avoiding the degradation of clustering quality caused by the merging operation. If and only if clusterA and clusterB meet these two criteria, the system will merge them into a new cluster clusterM. After the merger, clusterM will contain all matching objects in clusterA and clusterB at the same time. At the same time, the system deletes clusterA and clusterB from clusterList, adds clusterM, and completes the update of the cluster. Repeat the process to merge similar clusters. Each round of merging will reduce the number of clusters in clusterList and increase the similarity of matching objects within the cluster. When the entire similarity matrix simMatrix is ​​traversed and no more cluster pairs that meet the merging conditions can be found, the clustering process ends.At this point, the clusters in clusterList have reached a relatively stable state, the matching objects in each cluster are highly similar, and the distinction between clusters is high. The final cluster list clusterList is output as the result of clustering the matching result objects. Each cluster cluster in clusterList represents a group of matching fragments that are highly similar to each other but have low similarity to other groups. This cluster-optimized matching result organization form can more accurately and clearly reflect the repeated patterns and internal connections between the procurement documents to be compared.

[0064] After the system completes the clustering of the matching result objects, it needs to further process each cluster in the cluster list clusterList to extract the differences. Through difference extraction, the system can highlight the personalized expressions of different sub-documents while retaining the common content of the matching fragments, making the preliminary comparison results more comprehensive and detailed.

[0065] The specific steps of difference extraction are as follows: For each cluster cluster in clusterList, the system first extracts all the matching result objects contained therein, and sorts them from large to small according to the length of the matching text. The longer the text length, the richer the information contained in the matching fragment, and the more likely it is to cover the core content of the cluster. By sorting by length, the system can give priority to the matching fragments with the most complete content. Select the matching result object with the largest matching length in cluster as the benchmark object baseMatch. The benchmark object is usually the best representative of the cluster, containing the main content and key information of the cluster. Subsequent difference extraction will use baseMatch as the reference system.

[0066] Traverse other matching objects match in cluster and find out the differences between them and the benchmark object baseMatch: use a sequence alignment algorithm (such as the Needleman-Wunsch algorithm) to align the text content of match and baseMatch and extract their longest common subsequence LCS. The longest common subsequence represents the maximum common content of the two matching objects and is a reflection of their similarity. Specifically, initialize a two-dimensional matrix dp. The number of rows in the matrix is ​​the length of the two character sequences to be compared plus 1, and the number of columns is the length of the two character sequences to be compared plus 1. Each element dp[i][j] in the matrix represents the length of the longest common subsequence of the first i characters of sequence A and the first j characters of sequence B. Fill the first row and first column of the matrix dp. Since the LCS length of the empty sequence and any sequence is 0, dp[0][j] and dp[i][0] are both initialized to 0. Traverse the matrix dp and fill each element dp[i][j] row by row. The filling rules are as follows: If the i-th character of sequence A is equal to the j-th character of sequence B, then dp[i][j]=dp[i-1][j-1]+1, that is, add 1 to the LCS length of A[0...i-1] and B[0...j-1]. If the i-th character of sequence A is not equal to the j-th character of sequence B, then dp[i][j]=max(dp[i-1][j], dp[i][j-1]), that is, take the larger value between the LCS length of A[0...i-1] and B[0...j] and the LCS length of A[0...i] and B[0...j-1]. After filling the matrix dp, dp[i][j] stores the length of the longest common subsequence of sequence A and sequence B. In order to obtain the specific content of the longest common subsequence, it is necessary to backtrack the dp matrix. The backtracking process starts from dp[i][j], and recursively determines each character in the LCS based on the decision made during the table filling process: If A[i]=B[j], add the character to the LCS and jump to dp[i-1][j-1] to continue backtracking. If A[i]!=B[j], compare the size of dp[i-1][j] and dp[i][j-1], jump to the position corresponding to the larger value and continue backtracking. When i or j is equal to 0, the backtracking terminates and the character sequence of the LCS is output in reverse order. The system takes the text content of the two matching result objects match and baseMatch to be compared as character sequences A and B respectively, and applies the Needleman-Wunsch algorithm to extract their longest common subsequence LCS. By tracking the common characters in the two sequences, the algorithm can find the maximum shared content of match and baseMatch in semantics and expression, revealing their inherent similarities. At the same time, the time complexity of the algorithm is O(mn), where m and n are the lengths of the sequences to be compared. It has high operating efficiency and is suitable for the comparison needs of large-scale text data.

[0067] After extracting the longest common subsequence LCS of the two matching result objects match and baseMatch, the system needs to further identify and mark their difference parts. Specifically, the system marks the text fragments in match that are not covered by LCS as the difference part diffText1, and marks the text fragments in baseMatch that are not covered by LCS as the difference part diffText2. These difference parts represent the uniqueness of match and baseMatch in terms of expression, reflecting the personalized description of the same thing by different sub-documents, and are important information that cannot be ignored in comparison analysis. By comparing the starting position of diffText1 in match with the starting position of LCS in match, it can be determined whether diffText1 is located at the front or back of LCS. If the starting position of diffText1 is less than the starting position of LCS, it means that diffText1 is located at the front of LCS, that is, diffText1 appears in the first half of match. If the starting position of diffText1 is greater than the ending position of LCS, it means that diffText1 is located at the back of LCS, that is, diffText1 appears in the second half of match. According to the relative position of diffText1, the system creates the corresponding difference match object subMatch: If diffText1 is located at the front of LCS, a difference match object subMatch1 is created, and its text content is set to diffText1. The starting position and ending position in the main document are the same as the starting position and ending position of diffText1 in match. If diffText1 is located at the back of LCS, a difference match object subMatch2 is created, and its text content is set to diffText1. The starting position and ending position in the main document are the same as the starting position and ending position of diffText1 in match. If the starting position and ending position of diffText1 in match coincide with the starting position and ending position of LCS respectively, it means that match and baseMatch are exactly the same, there is no difference, and there is no need to create a subMatch object. Through the above difference positioning and marking process, the system can accurately extract the difference fragments of match and baseMatch, and record their specific positions in the original main document through the subMatch object. This approach ensures the accuracy and traceability of the difference information, and provides a reliable basis for the subsequent generation of high-quality comparison reports. Compared with the traditional text comparison method based on keywords or grammatical rules, the LCS difference extraction technology of this application is more sophisticated and comprehensive. It fully considers the relative position relationship of the text fragments in the original document and realizes the precise positioning and description of the difference information by creating independent difference matching objects.This not only improves the accuracy of difference extraction, but also facilitates the visualization and interactive analysis of difference information. Users can intuitively see the distribution of difference segments in the document and conduct targeted in-depth analysis of specific segments as needed.

[0068] Add the newly created subMatch object to the subsidiary object list subMatchList of the base object baseMatch as a supplementary description of baseMatch. The difference fragments in the subsidiary list, together with the base match, constitute the complete repeated content. After traversing all the matching objects in cluster, the system adds the base object baseMatch together with its subsidiary object list subMatchList as a merged matching result object to the merged matching result list mergedMatchList. This merged matching result object fully describes a repeated content fragment, which contains both the common content of each sub-document and reflects their individual differences. Repeat the execution and process all clusters in clusterList in turn until all the difference information is extracted. After processing all clusters, the system sorts them in ascending order according to the starting position of each object in the merged matching result list mergedMatchList in the main document to obtain the preliminary comparison results prelimResults of the group. This ordered prelimResults list is the result of the overall comparison analysis of the group. Each element in the list consists of a base match object baseMatch and several subsidiary difference objects subMatch, reflecting the similarities and differences of a repeated segment in the main document in each sub-document.

[0069] S7, after the system generates the preliminary comparison results, it needs to further optimize and process them to identify and remove the matching content reported repeatedly, so as to obtain the final comparison results. This process can improve the accuracy and readability of the comparison results and avoid the interference of redundant information on user analysis and judgment. Specifically, the system traverses the list of preliminary comparison results prelimResults of a certain group, and accesses each matching result object prelimMatch in turn. For each prelimMatch, the system performs the following operations: extract the key information of the matching result object prelimMatch: the text content of the repeated content: prelimMatch.content; the starting position of the repeated content in the main document: prelimMatch.mainStart; the ending position of the repeated content in the main document: prelimMatch.mainEnd; b. Search in the final comparison result list finalResults whether there is a matching object finMatch that is repeated with the current matching result object prelimMatch. The criteria for judging duplication are: finMatch.content is the same as prelimMatch.content, that is, the text of the duplicate content is exactly the same; finMatch.mainStart is the same as prelimMatch.mainStart, that is, the starting position of the duplicate content in the main document is the same; finMatch.mainEnd is the same as prelimMatch.mainEnd, that is, the ending position of the duplicate content in the main document is the same; if a duplicate matching object finMatch that meets the above conditions is found in finalResults, go to step 2; if no duplicate object is found, go to step 3. Identify that the currently traversed matching result object prelimMatch is a duplicate reported matching content, filter it out, and do not add it to the final comparison result finalResults. This can effectively remove redundant matching items in the preliminary results.

[0070] If the currently traversed matching result object prelimMatch does not find duplicates in finalResults, it will be identified as a new, non-duplicate matching content and needs to be added to the final comparison result list finalResults: a. Create a new matching object finalMatch, and copy the various information of the matching result object prelimMatch to finalMatch, including the text of the duplicate content, the starting position and the ending position in the main document, etc. b. Extract the position information and document number of the duplicate content in each sub-document from the subsidiary object list subMatches of the matching result object prelimMatch: extract the starting position of the duplicate content in each sub-document subDoc, and add them to the finalMatch.subStarts list in turn; extract the ending position of the duplicate content in each sub-document subDoc, and add them to the finalMatch.subEnds list in turn; extract the document number of each sub-document subDoc containing the duplicate content, and add them to the finalMatch.subDocs list in turn; c. Add the matching object finalMatch that integrates the complete information to the final comparison result list finalResults. After traversing all the matching result objects in the prelimResults list, sort the finalResults list in ascending order according to the starting position of the matching object in the main document, finalMatch.mainStart, to obtain the final comparison result list of the group. In the final result, the duplicate contents of the same group will be listed in the order of their occurrence in the main document.

[0071] At this point, the system has completed the deduplication optimization of the preliminary results and the generation of the final results. Through the above processing, the final output comparison results can present all the repeated content fragments in the document group to the user in a concise, orderly and non-redundant manner, and clearly identify the specific location of each fragment in the main document and the sub-document. This form of organization of comparison results is highly logical and readable, which is convenient for users to quickly locate and analyze the duplication problems of document groups. Unlike direct full-text comparison, this application makes full use of the intermediate results of clustering and difference analysis, continuously improves and refines the matching information of repeated content, and through necessary redundant filtering, maximizes the focus and accuracy of the final comparison results. This progressive and hierarchical comparison result generation mechanism improves the comparison efficiency.

[0072] S8, after generating the final comparison results of each group, the system also needs to visually mark the repeated content in the main document of the corresponding group to generate an intuitive and easy-to-read result document. This marking process can make full use of the existing functions of the word processing software such as the revision mode and highlight background, flexibly highlight the repeated fragments of the document, and facilitate users to quickly locate and identify. The specific steps of generating the result document and marking the repeated content are as follows: the system traverses the final comparison results finalResults of each group group obtained in step S7, extracts the main document mainDoc corresponding to the current group, and uses it as the target document of the marking operation. For the final comparison results finalResults of the current group, the system traverses each matching result object finalMatch therein, and locates the repeated content fragment indicated by the matching object in the main document mainDoc: the following key position information is obtained from the matching result object finalMatch: the starting position of the repeated content in the main document mainDoc: finalMatch.mainStart; the ending position of the repeated content in the main document mainDoc: finalMatch.mainEnd; according to the starting position finalMatch.mainStart and the ending position finalMatch.mainEnd of the repeated content, the text area where the content is located is accurately located in the main document mainDoc. The system can use the API provided by the word processing software or a custom string processing function to implement this positioning process. For each repeated content text area located, the system uses the revision mode to mark it: the text in the repeated content text area is set to the deleted state and displayed in the style of a strikethrough, which intuitively indicates that the content fragment is repeated in the main document and can be deleted. A revision annotation is added below the repeated content text area, and the content of the annotation is the relevant information of all sub-documents subDoc matched by the repeated fragment. The steps to generate this annotation are:

[0073] Get the sub-document list recorded in the matching result object finalMatch: finalMatch.subDocs; traverse each sub-document subDoc in the sub-document list finalMatch.subDocs: get the document number of the current sub-document subDoc; get the starting position and ending position of the repeated content fragment in the current sub-document subDoc; concatenate the above information (sub-document number, repeated content position) in the agreed format as the annotation content of a single sub-document; after traversing all sub-documents, concatenate the annotation contents of each sub-document according to the agreed delimiter as the complete annotation content of the repeated content area; through the above revised annotations, users can clearly see in which other sub-documents a repeated fragment also appears, and their specific positions in these sub-documents.

[0074] For each duplicate content text area located, the system also uses a highlighted background to mark it: set a striking highlighted background color to highlight the duplicate content area, and set the background color of the duplicate content text area to the above-mentioned highlighted background color. Adding a highlighted background can further enhance the recognition of the duplicate content area, making it more conspicuous in the entire document. After traversing all matching result objects in the final comparison result finalResults, the system generates a marked result document resultDoc based on the main document mainDoc: copy the main document mainDoc to generate a new result document resultDoc; apply the duplicate content annotation revision (deletion line, annotation, highlight background) to the corresponding position in the result document resultDoc, so that the annotation information takes effect in the new document; the system associates the marked result document resultDoc with the corresponding group group and the ID of this duplicate check request as the final output comparison result document of the group. The result document can be saved in a format supported by word processing software (such as DOCX, PDF, etc.) for direct viewing by users.

[0075] S9, after receiving the document set to be compared submitted by the user, the system needs to generate a unique request ID for identifying and managing the duplicate checking task of the batch of documents. This request ID can help users accurately query and track the progress and results of the comparison task they submitted, and also facilitate the system to schedule and persist the task. The specific request ID generation includes: the system uses the UUID (Universally Unique Identifier) ​​algorithm based on timestamp, hardware identification code and random number to generate a 32-bit request ID string as the global unique identifier of the batch of documents to be compared. The UUID algorithm can ensure the uniqueness and unpredictability of the ID generated in a distributed environment. It constructs a 160-bit binary number by combining the timestamp, hardware identification code (such as MAC address, CPU serial number, etc.) and random number, and then converts it into a 32-bit hexadecimal string. This string has strong randomness and uniqueness, and is very suitable as a request ID for asynchronous tasks. The system associates the generated request ID with the storage path or URL of the N documents to be compared received in step S1 to build a request context structure. This structure records the correspondence between the request ID and the specific document resource, so that subsequent tasks can accurately locate the relevant input document according to the request ID. The request context structure can be stored in a variety of forms such as relational databases, key-value databases, JSON files, etc. The system persists the request context structure in a dedicated request context database to ensure that the request information is still available after the server restarts or fails. The system returns the generated request ID string to the user who submitted the comparison request through the network interface (such as HTTP / HTTPS protocol) as the only credential for the user to retrieve the comparison task progress and final output results. After receiving the request ID, the user can regularly send progress query requests to the system. The system finds the corresponding task record according to the request ID and returns the current processing progress (such as "submitted", "parsing documents", "extracting semantic vectors", "clustering and grouping", etc.). When the task is finally completed, the user uses the request ID to query again, and the system returns a formatted comparison result report.

[0076] S10, when the system completes all processing steps of the document set corresponding to the request ID, it will automatically generate an overall comparison report based on the comparison results of each group generated in step S7. The report uses the system preset report template to summarize and count the comparison results of multiple groups, and mainly includes the following contents: Basic document data: Lists the number, name, size and other meta information of the original documents submitted by the user, helping users to quickly understand the input data overview of the comparison task. Overall document repetition rate: Calculates and displays the overall average repetition rate of all documents, intuitively reflecting the overall repetition degree of the document set. In-group repetition rate: Counts the average repetition rate, highest repetition rate, number of duplicate documents and other indicators in each group, helping users to find and locate highly similar documents. Duplicate content distribution: Displays the distribution of the number of documents in different repetition rate ranges through visual charts such as pie charts and bar charts, allowing users to intuitively grasp the overall distribution of duplicate problems. Full text comparison: Attaches the annotation results of each group main document generated in step S8 to the report, which is convenient for users to directly view the distribution and source of duplicate content in the document, and conduct manual verification and judgment. The comparison report can be generated in common formats such as PDF, HTML, DOCX, etc. to ensure that it can be viewed normally on different devices and system environments.< / sparseelement>

Claims

1. A document comparison method based on an improved LCS algorithm, characterized in that: include: S1, receiving N documents to be compared uploaded by the user, the upload path is a local document path, a network URL or a document server; the document type is Microsoft Word format; S2, divide the N documents into N groups, each group contains a main document and N-1 secondary documents; wherein the main document refers to the first document of the corresponding group; the secondary document refers to the remaining documents of the corresponding group except the main document; S3, setting a minimum common subsequence length threshold L0 for determining repeated content; receiving stop words preset by the user; S4, for each group, construct a virtual two-dimensional array, with the number of characters in the primary document as the number of rows and the number of characters in the secondary document as the number of columns, using sparse matrix storage to record only the matching character positions; Starting from the element in the last row and last column of the virtual two-dimensional array, traverse along the main diagonal direction of the two-dimensional array, row by row to the element in the first row and first column, and record the content of the continuous matching characters, the starting position and the ending position in the primary document and the secondary document; S5, multi-thread parallel processing is performed on the N groups generated by S2, and each thread calls step S4 to compare the primary document and the secondary document in a group; the comparison results within the group obtained by each thread are passed to step S6; S6, for each group, merging the comparison results of each secondary document in the corresponding group with the primary document to generate a preliminary comparison result including all duplicate contents; S7, identifying and removing the matching contents of the duplicate reports in the preliminary comparison results, and obtaining the final comparison results of the corresponding groups; wherein the final comparison results include the location of the duplicate contents in the main document and the secondary document of the source; S8, in each grouped main document, using the revision mode and the highlighted background to mark the repeated contents in the final comparison result generated in step S7, and generating a marked result document; S9, generating a 32-bit UUID as a request ID for the user to query the comparison progress and comparison results; returning the request ID to the user, and retrieving the corresponding comparison progress and results according to the request ID; S10, generating a comparison report according to the comparison results of each group generated in step S7, the comparison report including basic data of the document, overall repetition rate of the document, repetition rate within the group, distribution of repetitive content and full text comparison.

2. The document comparison method based on the improved LCS algorithm according to claim 1, characterized in that: S2, divide N documents into N groups, each group contains a main document and N-1 sub-documents, including: According to the serial numbers or submission time order of the N documents to be compared, the i-th document is used as the primary document of the i-th group, where i is a positive integer; for each group, all documents except the primary document are used as secondary documents of the corresponding group; According to the grouping results, create a list groups consisting of N groups, each group contains a main document and N-1 sub-documents; Traverse N documents, take the i-th document as the main document, create a new grouping object group; encapsulate the i-th document into the main document object mainDoc, and add it to the group; Traverse N documents. When j is not equal to i, encapsulate the jth document as a sub-document object subDocs and add it to the sub-document object list of group. Add the group object group to the group list groups; For each group in the group list groups, extract the main document object mainDoc and the sub-document object subDocs list, and construct a dictionary structure groupdict containing the main document path and the sub-document path list; Associate the dictionary structure groupdict with the request ID.

3. The document comparison method based on the improved LCS algorithm according to claim 2, characterized in that: S3, setting the minimum common subsequence length threshold L0, including: Count the average paragraph length of each document to be compared; Sort the array arrAvgParaLen consisting of the average paragraph lengths of all documents, and use the middle value of the sorted array arrAvgParaLen as the median medianLen: medianLen = arrAvgParaLen[N / 2]; Set the initial value k0 of the threshold coefficient and the initial value L00 of the minimum common subsequence length threshold L0: L00 = k0 *medianLen; The minimum common subsequence length threshold L0 is set according to k0 and L00.

4. The document comparison method based on the improved LCS algorithm according to claim 3, characterized in that: The minimum common subsequence length threshold L0 is set according to k0 and L00, using the following formula: L0 = k * medianLen Where k is the threshold coefficient; k = k0 + λ*simThresh Among them, λ is the adjustment coefficient, ranging from 0.1 to 0.5; simThresh is the threshold of the similarity score, ranging from 0.6 to 0.

9.

5. The document comparison method based on the improved LCS algorithm according to claim 3, characterized in that: S4, for each group, construct a virtual two-dimensional array, including: For each group in the group list groups, get the main document mainDoc and the sub-document list subDocs; For the main document mainDoc and each sub-document subDoc in the sub-document list subDocs, construct a virtual two-dimensional array matrix: The number of characters in the main document mainDoc is used as the number of rows m in the virtual two-dimensional array matrix; The number of characters in the sub-document subDoc is used as the number of columns n of the virtual two-dimensional array matrix; A sparse matrix is ​​used to store a virtual two-dimensional array matrix, which only records the positions of matching characters in the main document mainDoc and the sub-document subDoc; Starting from the element matrix[m-1][n-1] in the last row and last column of the virtual two-dimensional array matrix, traverse along the main diagonal direction of the two-dimensional array matrix row by row to the element matrix[0][0] in the first row and first column; Record the content of consecutive matching characters and the starting and ending positions of the corresponding content in the main document mainDoc and the sub-document subDoc, and construct the matching result object match: Record the continuously matched character content as match.content; The starting position of the matching content in the main document mainDoc is recorded as match.main_start; The end position of the matching content in the main document mainDoc is recorded as match.main_end; The starting position of the matching content in the sub-document subDoc is recorded as match.sub_start; The end position of the matching content in the sub-document subDoc is recorded as match.sub_end; Add the matching result object match to the matching result list matchList as the matching result between the main document mainDoc and the current sub-document subDoc; After traversing all sub-documents in the sub-document list subDocs, the matching result list matchList is used as the matching result of the current group group.

6. The document comparison method based on the improved LCS algorithm according to claim 5, characterized in that: For the main document mainDoc and each sub-document subDoc in the sub-document list subDocs, construct a virtual two-dimensional array matrix, including: Perform structured parsing on the main document mainDoc and the sub-document subDoc to obtain key data, including tobacco variety, quantity, unit price, origin, and purchase time; Normalize the extracted key data; Construct a structured document object docObj based on the normalized key data; The structured document object constructed by the main document mainDoc is used as the row label of the virtual two-dimensional array matrix; the structured document object constructed by the sub-document subDoc is used as the column label of the virtual two-dimensional array matrix; Compare the corresponding fields of the primary document structured object and the secondary document structured object, and calculate the similarity of each field; The calculated similarity values ​​of each field are weighted averaged to obtain the overall similarity between the primary document structured object and the secondary document structured object, which is used as the value of the corresponding position in the virtual two-dimensional array matrix.

7. The document comparison method based on the improved LCS algorithm according to claim 5, characterized in that: S6, for each group, generates preliminary comparison results containing all duplicate contents, including: Extract all matching result objects from the group matching results to form a matching result object list matchList; Calculate the similarity between each matching result object match and other matching result objects in the matching result object list matchList; A similarity matrix simMatrix is ​​constructed according to the similarity, and the row and column indexes correspond to the positions of the matching result objects in the matchList; wherein the rows and columns of the similarity matrix simMatrix correspond to the matching result objects match in the matching result object list matchList, and the element values ​​of the similarity matrix simMatrix correspond to the similarity between the two matching result objects match; the similarity matrix simMatrix is ​​a symmetric matrix; Cluster the matching result objects according to the similarity matrix simMatrix to obtain the cluster list clusterList; For each cluster cluster in the cluster list clusterList, extract all matching result objects in the cluster cluster and sort them from large to small according to the matching length; Select the matching result object with the largest matching length as the base object baseMatch; Traverse the matching objects in the cluster cluster and extract the differences between the matching objects and the benchmark object baseMatch: Calculate the difference between the matching object and the reference object in terms of content and position; Extract the difference part as an independent matching result object and add it to the list of subsidiary objects of baseMatch; Add the base object baseMatch and the corresponding list of subordinate objects as the merged match result objects to the merged match result list mergedMatchList; After traversing all clusters, the matching result objects in the merged matching result list mergedMatchList are sorted according to their starting positions in the main document to obtain the preliminary comparison results prelimResults of the corresponding groups.

8. The document comparison method based on the improved LCS algorithm according to claim 7, characterized in that: Cluster the matching result objects according to the similarity matrix simMatrix to obtain the cluster list clusterList, including: Each matching result object match in the matching result object list matchList is used as the initial cluster cluster, and each cluster contains only one matching object; Traverse the similarity matrix simMatrix and obtain the two clusters clusterA and clusterB with the largest current similarity simAB; Calculate the average similarity meanSimA between the matching objects in cluster clusterA; Calculate the average similarity meanSimB between the matching objects in cluster clusterB; If the similarity simAB is greater than the preset similarity threshold simThreshold, and is greater than meanSimA and meanSimB at the same time, clusters clusterA and clusterB are merged to obtain a new cluster clusterM; Repeat the merging steps until there are no cluster pairs that meet the merging conditions in the similarity matrix simMatrix; Output the cluster list clusterList as the result of clustering the matching result objects.

9. The document comparison method based on the improved LCS algorithm according to claim 8, characterized in that: Traverse the matching objects in the cluster cluster and extract the differences between the matching objects and the benchmark object baseMatch: Calculate the difference between the matching object and the reference object in terms of content and position, including: Initialize the subsidiary object list subMatchList of the base object baseMatch to an empty list; Traverse each matching object match in the cluster cluster, record the text content of the matching object match as matchText, and record the text content of the base object baseMatch as baseText; The Needleman-Wunsch algorithm is used to extract the longest common subsequence LCS of matchText and baseText; The substring in matchText that is not included in LCS is taken as the difference part diffText1; The substrings in baseText that are not included in LCS are taken as the difference part diffText2; If diffText1 is at the front of matchText, a new match result object subMatch1 is created, whose starting position is the starting position of match and whose ending position is the ending position of diffText1; If diffText1 is at the end of matchText, a new match result object subMatch2 is created, whose starting position is the starting position of diffText1 and the ending position is the ending position of match; The extracted independent matching result object subMatch is added to the subsidiary object list subMatchList of the base object baseMatch.

10. The document comparison method based on the improved LCS algorithm according to any one of claims 2 to 9, characterized in that: S9, generates a 32-bit UUID as the request ID, including: A UUID algorithm based on timestamp, hardware identification code and random number is used to generate a 32-bit request ID string as the unique identifier of the document set to be compared; Associating the generated request ID with the N local document paths or URLs received in step S1, constructing a request context structure, and storing it in a request context database; The generated request ID string is returned to the user who submitted the comparison request through the network interface, and serves as the only credential for the user to subsequently retrieve the comparison task progress and final output results.

Images