Bidding document duplicate checking method, system and equipment and medium

Abstract

The invention provides a bidding document duplicate checking method, system and device and a medium. The method comprises the steps of obtaining file content of a bidding document; the bidding document comprises a document to be subjected to duplicate checking and a comparison document; extracting sub-strings of the file content; performing bucket dividing processing on the sub-character strings, and mapping the sub-character strings meeting a preset condition into the same hash bucket; calculating the similarity between the sub-character strings of the document to be subjected to duplicate checking and the sub-character strings of the comparison document in the same hash bucket, and determining the pairing relationship between the sub-character strings; performing pairing and merging according to the pairing relationship to obtain a maximum repeated fragment; and calculating a content repetition rate according to the maximum repetition fragment so as to determine a duplicate checking result of the bidding document. Similar sub-strings are concentrated in the same hash bucket through a bucket dividing strategy, so that rapid repeated detection of large-scale text data is realized, and the duplicate checking response speed is increased; and determining the maximum duplicate fragment according to the similarity and the pairing relationship, thereby solving the problem that duplicate fragments with a small number of altered characters cannot be identified.

Description

Technical Field

[0001] This application relates to the field of aerospace remote sensing technology, and in particular to a method, system, equipment and medium for checking the duplication of tender documents. Background Technology

[0002] In the field of bid document management, bid document plagiarism detection is an important technology to ensure the originality of content and prevent bid rigging and collusion. Current mainstream technologies typically follow a process of "preprocessing-feature extraction-similarity calculation": first, the text is standardized through operations such as stop word filtering and word segmentation, and then a vector space model combined with a cosine similarity algorithm is used to determine duplicates.

[0003] However, when dealing with large-scale long texts, the system response speed of this method often fails to meet the real-time detection requirements, and it lacks character-level repetition location capabilities, making it impossible to accurately identify the specific intervals of repeated content. This causes inconvenience for users' review and proofreading, and it is difficult to adapt to large-scale text scenarios. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, electronic device, and storage medium for checking the duplication of tender documents, which can solve the problems of slow response speed, inability to accurately identify duplicate content, and difficulty in adapting to large-scale text scenarios in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for detecting plagiarism in tender documents, the method comprising: Obtain the content of the bidding documents; the bidding documents include the document to be checked for plagiarism and the comparison document. Extract substrings from the file content; The substrings are bucketed, and substrings that meet the preset conditions are mapped to the same hash bucket; Calculate the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and determine the pairing relationship between the substrings; Based on the pairing relationship, the pairs are merged to obtain the maximum repeating segment; The content duplication rate is calculated based on the maximum duplicate segment to determine the plagiarism detection result of the tender document.

[0006] Optionally, the substring includes a first substring, and the substring for extracting the file content includes: The first window is slid across the file content character by character according to its size, extracting the string within each window until the entire file content has been traversed, thus obtaining the first substring; the first window size represents the size of the string that can be extracted from the first window.

[0007] Optionally, the substring further includes a second substring, and the pairing and merging based on the pairing relationship to obtain the maximum repeating segment includes: The first window size is expanded, and the second window is slid through the file content character by character according to the second window size to extract the string in each second window, thus obtaining the second substring; the second window size represents the size of the string that can be extracted from the second window.

[0008] Optionally, the step of bucketing the substrings and mapping substrings that meet preset conditions to the same hash bucket includes: Extract the feature vector of the substring; The obtained feature vector is hashed using the MinHash function to obtain the corresponding signature vector; The substrings are bucketed according to the signature vector, and substrings whose signature vectors meet preset conditions are mapped to the same hash bucket.

[0009] Optionally, the step of bucketing the substrings according to the signature vector, mapping substrings whose signature vectors meet preset conditions to the same hash bucket, includes: The signature vector is divided into a preset number of bands; A hash function is used to process the components of each band, generating a corresponding hash bucket identifier; If the signature vector corresponding to the substring of the document to be checked and the signature vector corresponding to the substring of the comparison document meet the preset conditions in at least one band, then the substring corresponding to the signature vector is mapped to the same hash bucket.

[0010] Optionally, calculating the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and determining the pairing relationship between the substrings, includes: Calculate the text character deviation distance between the substring of the document to be checked and the substring of the comparison document in the same hash bucket; The similarity between the substrings of the document to be checked and the substrings of the comparison document is calculated based on the text character deviation distance. The pairing relationships between substrings with a similarity greater than or equal to the similarity threshold are stored according to a preset data structure, and the substrings with a similarity greater than or equal to the similarity threshold are set as similar substring pairs.

[0011] Optionally, the preset data structure includes: the file identifier of the tender document to which the substring in the similar substring pair belongs, the starting position of the substring in the similar substring pair, and the ending position of the substring in the similar substring pair.

[0012] Optionally, the step of pairing and merging according to the pairing relationship to obtain the maximum repeating segment includes: Based on the pairing relationship, adjacent similar substring pairs are determined; Based on the preset data structure of the adjacent similar substring pairs, it is detected whether the substrings contained in the similar substring pairs meet the merging condition in both the document to be checked and the comparison document; If the merging conditions are met, adjacent similar substring pairs are merged into the largest repeating segment.

[0013] Optionally, the step of calculating the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document includes: The documents are merged based on the largest duplicate segment to obtain a merged document; Delete the preset template content from the merged document; Calculate the content duplication rate of the merged document after deleting the preset template content based on the maximum duplicate segment; The plagiarism detection result of the document to be checked is determined based on the content repetition rate.

[0014] Secondly, embodiments of this application provide a system for detecting plagiarism in tender documents, the system comprising: The tender document acquisition module is used to acquire the file content of the tender documents; the tender documents include the document to be checked for plagiarism and the comparison document; The substring extraction module is used to extract substrings from the file content; The bucketing module is used to bucket the substrings and map substrings that meet preset conditions to the same hash bucket. The pairing relationship determination module is used to calculate the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and to determine the pairing relationship between the substrings; The merging module is used to perform pairing and merging based on the pairing relationship to obtain the maximum repeating segment; The plagiarism detection result generation module calculates the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0017] Compared with the prior art, the embodiments of this application have the following advantages: In this embodiment, the content of a bidding document is obtained; the bidding document includes a document to be checked for plagiarism and a comparison document; substrings of the document content are extracted; the substrings are bucketed, and substrings that meet preset conditions are mapped to the same hash bucket; the similarity between the substrings of the document to be checked and the substrings of the comparison document in the same hash bucket is calculated, and the pairing relationship between the substrings is determined; the pairing is merged according to the pairing relationship to obtain the maximum duplicate segment; the content duplication rate is calculated based on the maximum duplicate segment to determine the plagiarism detection result of the bidding document. By using a bucketing strategy to concentrate similar substrings in the same hash bucket, fast duplication detection of large-scale text data is achieved, improving the plagiarism detection response speed; determining the maximum duplicate segment based on similarity and pairing relationship solves the problem of not being able to identify duplicate segments with a small number of character modifications. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of a method for checking the duplication of tender documents provided in an embodiment of this application; Figure 2 This is a structural block diagram of a bid document plagiarism detection system provided in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application; Figure 4 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The following description, in conjunction with the accompanying drawings, details a method, system, electronic device, and storage medium for checking the duplication of tender documents provided in this application, through specific embodiments and application scenarios.

[0022] In the field of bid document management, bid document plagiarism detection is an important technology to ensure the originality of content and prevent bid rigging and collusion. Current mainstream technologies typically follow a process of "preprocessing-feature extraction-similarity calculation": first, the text is standardized through operations such as stop word filtering and word segmentation, and then a vector space model combined with a cosine similarity algorithm is used to determine duplicates.

[0023] However, from a technical performance perspective, existing methods suffer from significantly increased computational complexity when handling multi-text cross-comparison due to the rising dimensionality of feature vectors, especially when dealing with large-scale, long texts. The system response speed often falls short of real-time detection requirements. From an accuracy perspective, current technologies largely rely on document-level similarity assessment, lacking character-level duplicate location capabilities and failing to accurately identify specific intervals of duplicate content, thus hindering user review and proofreading. From a results presentation perspective, current systems generally suffer from visualization deficiencies; crucial dimensions such as the contextual relationships of duplicate content and cross-file source paths are difficult to display intuitively, affecting users' complete understanding of the duplicate context. These issues include slow plagiarism detection response speed, inability to accurately identify duplicate content, and difficulty adapting to large-scale text scenarios.

[0024] Reference Figure 1 The diagram illustrates a flowchart of a method for detecting plagiarism in tender documents provided in an embodiment of this application, which may specifically include the following steps: Step 101: Obtain the file content of the bidding documents; the bidding documents include the document to be checked for plagiarism and the comparison document; In this embodiment of the application, it is necessary to obtain the file content of the document to be checked for plagiarism and the comparison document. Since the bidding documents have different formats, different methods will be used to obtain their file content for bidding documents of different formats.

[0025] In the implementation, for different formats of bidding documents, corresponding third-party libraries are used to read the document content, including paragraphs and tables. A preprocessing stage then follows, cleaning the read bidding documents by using regular expressions to remove headers and footers, redundant whitespace characters, and special symbols, ensuring accuracy and consistency in subsequent processing. It is important to note that the original line breaks are preserved during preprocessing to ensure that paragraph formatting remains largely consistent with the original material when identifying duplicate content, making it easier for users to review and proofread.

[0026] Step 102: Extract substrings from the file content; In practical implementation, a sliding window technique can be used to extract substrings from the file content until the entire document has been traversed. A substring library is then built based on the extracted substrings. The substring library only records the position of the substring and the unique identifier information of the original bidding document in which it is located, thereby saving a lot of memory space.

[0027] Step 103: Perform bucketing on the substrings, mapping substrings that meet the preset conditions to the same hash bucket; Since different substrings have different feature vectors, this embodiment can use a word embedding-based vector representation method to process the substrings after extracting the substrings of the file content, and perform bucketing on the substrings, mapping substrings that meet the preset conditions to the same hash bucket, which lays the foundation for the rapid search of similar substrings in the future.

[0028] Step 104: Calculate the similarity between the substrings of the document to be deduplicated and the substrings of the comparison document in the same hash bucket, and determine the pairing relationship between the substrings; After bucketing the substrings, the similarity between the substrings of the document to be checked and the substrings of the comparison document in the same hash bucket is calculated, and the pairing relationship between the substrings of the document to be checked and the substrings of the comparison document is established based on the similarity.

[0029] Step 105: Pair up and merge according to the pairing relationship to obtain the maximum repeating segment; In real-world scenarios, there may be overlapping content between multiple substrings, and / or situations where consecutively repeated content exceeds the window size for extracting substrings. Therefore, in this embodiment, after obtaining the pairing relationship between the substrings of the document to be checked and the substrings of the comparison document, adjacent substrings are paired and merged from the beginning of the original document according to the pairing relationship until the end of the original document is reached, and substrings that meet the preset merging conditions are merged into the largest repeated segment.

[0030] Step 106: Calculate the content duplication rate based on the maximum duplicate segment to determine the duplication result of the tender document.

[0031] In this embodiment, the content duplication rate between the document to be checked and the comparison document is calculated based on the maximum duplicate segment. Documents to be checked with a content duplication rate greater than or equal to the duplication rate threshold are marked as duplicate documents, and a duplication report is generated as the duplication result. The duplication report and the duplicate content comparison table are integrated and displayed in HTML format. The duplicate content comparison table uses a list format to visually annotate the maximum duplicate segment between the document to be checked and the comparison document, highlighting the duplicate content and its corresponding context, which makes it easier for duplication checkers to quickly locate the duplicate position later.

[0032] In this embodiment, the content of a bidding document is obtained. The bidding document includes a document to be checked for plagiarism and a comparison document. Substrings of the document content are extracted. The substrings are bucketed, and substrings that meet preset conditions are mapped to the same hash bucket. The similarity between the substrings of the document to be checked and the substrings of the comparison document in the same hash bucket is calculated, and the pairing relationship between the substrings is determined. The pairing relationship is used to merge the substrings to obtain the maximum duplicate segment. The content duplication rate is calculated based on the maximum duplicate segment to determine the plagiarism detection result of the bidding document. By using a bucketing strategy to concentrate similar substrings in the same hash bucket, fast duplication detection of large-scale text data is achieved, improving the plagiarism detection response speed. Determining the maximum duplicate segment based on similarity and pairing relationship solves the problem of not being able to identify duplicate segments with only a few character modifications.

[0033] In one embodiment of this application, the substring includes a first substring, and the substring for extracting the file content includes: The first window is slid across the file content character by character according to its size, extracting the string within each window until the entire file content has been traversed, thus obtaining the first substring; the first window size represents the size of the string that can be extracted from the first window.

[0034] In the implementation, a sliding window technique is used to extract substrings from the file content. First, a minimum window size is given and set as the first window size. Then, the first window slides character by character across the file content according to this first window size, extracting the string within each window, until the entire file content has been traversed, yielding the first substring. Here, the first window size represents the size of the string that can be extracted within the first window. Using the sliding window technique to extract indexed substrings effectively solves the problem of unrecognized specialized vocabulary.

[0035] In one embodiment of this application, the substring further includes a second substring, and before performing pairing and merging according to the pairing relationship to obtain the maximum repeating segment, the method further includes: The first window size is expanded, and the second window is slid through the file content character by character according to the second window size to extract the string in each second window, thus obtaining the second substring; the second window size represents the size of the string that can be extracted from the second window.

[0036] In this embodiment, the substring also includes a second substring. Before pairing and merging according to the pairing relationship to obtain the maximum repeating segment, the size of the first window can be expanded by increasing the window size on the basis of the first window size to obtain the second window size. Then, the second window is slid character by character in the file content according to the second window size to extract the string in each second window to obtain the second substring.

[0037] In the specific implementation, the substring of the corresponding string size has already been obtained through the size of the first window. Since the length of professional words is not fixed, the substring extracted by the smallest window can be used as the basis, and the expansion can be carried out between the smallest window and the largest window with a single character as the step size, until the substring of the corresponding size is obtained through windows of different sizes. This further improves the accuracy of duplicate content recognition and avoids the problem of not being able to recognize due to the variable length of professional words.

[0038] In one embodiment of this application, the step of bucketing the substrings and mapping substrings that meet preset conditions to the same hash bucket includes: Extract the feature vector of the substring; The obtained feature vector is hashed using the MinHash function to obtain the corresponding signature vector; The substrings are bucketed according to the signature vector, and substrings whose signature vectors meet preset conditions are mapped to the same hash bucket.

[0039] In this embodiment, after obtaining the substring, the substrings of different sizes are sequentially subjected to feature extraction and transformation using a word embedding-based vector representation method to obtain the corresponding feature vectors, which are then mapped to the feature vector space. Subsequently, the MinHash function is used for hashing to generate the corresponding signature, resulting in a signature vector. Finally, the substrings are bucketed based on the signature vectors, and substrings whose signature vectors meet the preset conditions are mapped to the same hash bucket.

[0040] In its implementation, the MinHash function simulates random permutations using multiple hash functions to generate a fixed-length signature vector. Since the size of the substring extracted varies depending on the window size, and multiple comparison documents can exist for content duplication rate comparison with the document to be checked, thread pool technology can be used for concurrent processing across multiple comparison documents and their corresponding substring sizes, enabling fast duplication detection on large-scale text data.

[0041] This embodiment uses the MinHash function to obtain the corresponding signature vector. Based on the signature vector, bucketing can compress the features of substrings and reduce memory consumption. On the other hand, thread pool technology can be used to process multiple documents concurrently, enabling fast duplicate detection of large-scale text data.

[0042] In one embodiment of this application, the step of bucketing the substrings according to the signature vector and mapping substrings whose signature vectors meet preset conditions to the same hash bucket includes: The signature vector is divided into a preset number of bands; A hash function is used to process the components of each band, generating a corresponding hash bucket identifier; If the signature vector corresponding to the substring of the document to be checked and the signature vector corresponding to the substring of the comparison document meet the preset conditions in at least one band, then the substring corresponding to the signature vector is mapped to the same hash bucket.

[0043] In this embodiment, a locality-sensitive hashing (LSH) bucketing strategy is used to process the signature vector. The LSH bucketing strategy includes dividing each signature vector into b bands, each band containing r components, so that the total signature length is b×r.

[0044] In the specific implementation, for each band i, the hash function H_i is used to process the components of the band to generate its corresponding bucket identifier (bucket ID=H_i(band_i)). If the signature vector corresponding to the substring of the document to be checked and the signature vector corresponding to the substring of the comparison document have the same component value in at least one band, or are greater than or equal to the component threshold, then the feature vectors of the substring of the document to be checked and the substring of the comparison document corresponding to the signature vector are mapped to the same hash bucket.

[0045] This embodiment achieves the goal of mapping the feature vectors of similar substrings to the same hash bucket through the locality-sensitive hashing bucket strategy, thus laying the foundation for fast search of similar substrings in the future.

[0046] In one embodiment of this application, calculating the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and determining the pairing relationship between the substrings, includes: Calculate the text character deviation distance between the substring of the document to be checked and the substring of the comparison document in the same hash bucket; The similarity between the substrings of the document to be checked and the substrings of the comparison document is calculated based on the text character deviation distance. The pairing relationships between substrings with a similarity greater than or equal to the similarity threshold are stored according to a preset data structure, and the substrings with a similarity greater than or equal to the similarity threshold are set as similar substring pairs.

[0047] In this embodiment, after binning the substrings, the text character deviation distance between the substrings of the document to be checked and the substrings of the comparison document in the same hash bucket is calculated; then, the similarity between the substrings of the document to be checked and the substrings of the comparison document is calculated based on the text character deviation distance; the pairing relationship between substrings with similarity greater than or equal to the similarity threshold is stored according to a preset data structure, and the substrings with similarity greater than or equal to the similarity threshold are set as similar substring pairs.

[0048] In the implementation, since the feature vectors of similar substrings are already clustered in the same hash bucket, only substrings within the same hash bucket need to be compared to quickly find potentially similar substring pairs. This effectively reduces the number of substrings that need to be compared and improves the efficiency of plagiarism detection. Then, a custom similarity detection method is used to determine the degree of similarity between two substrings. This similarity is obtained by normalizing the text character deviation distance. The mathematical expression for the text character deviation distance is:

[0049] in, The text character deviation distance between two substrings S1 and S2, where S1 and S2 represent two substrings to be compared, and one of S1 and S2 is the substring extracted from the document to be checked for plagiarism. This represents the character at position i in substring S1. This represents the character at position i in substring S2. This is an indicator function that returns 1 if the characters at corresponding positions are different, and 0 if the characters at corresponding positions are different. For example, when the substrings S1="Three out of five people are male" and S2="Two out of seven people are male", since the values ​​of the 1st and 5th characters of the two strings are different, the text character deviation distance can be obtained by this formula = 2.

[0050] The mathematical expression for custom similarity is:

[0051] Where d is the text character deviation distance. ; d represents the size of the substring, i.e., the length of the substring, and s represents the normalized similarity, with a value between [0,1]. When d=0, s=1 indicates that the content of the two substrings is exactly the same, and the similarity between the substrings is the highest. When d=l, s=0 indicates that the content of the two strings is completely different, and the similarity between the substrings is the lowest.

[0052] The similarity calculated using the above formula is compared with a preset similarity threshold. If the similarity exceeds the preset similarity threshold, the substring blocks are considered to match. The pairing relationship between the two substrings is stored according to a preset data structure, and the two substrings are set as similar substring pairs. The above process is repeated until the pairing relationship of all substring sizes and their corresponding similar substring pairs are obtained.

[0053] In this embodiment, a custom similarity detection algorithm is used to calculate the similarity of substrings, which solves the problem of not being able to identify duplicate segments with a small number of character modifications, and records the corresponding pairing relationships, so that content exceeding the maximum window size can also be identified, thereby improving the accuracy of plagiarism detection.

[0054] In one embodiment of this application, the preset data structure includes: the file identifier of the tender document to which the substring in the similar substring pair belongs, the starting position of the substring in the similar substring pair, and the ending position of the substring in the similar substring pair.

[0055] In this embodiment of the application, the preset data structure includes: the file identifier of the tender document to which the substrings contained in the similar substring pair belong, the starting position of the substring, and the ending position of the substring.

[0056] In the specific implementation, the MatchPair data structure is used to store the pairing relationships. The contents of the MatchPair data structure are as follows: public class MatchPair { private int sourceFileId, sourceStart, sourceEnd; private int matchFileId, matchStart, matchEnd; } This data structure includes matchFileId, matchStart, and matchEnd; where matchFileId represents the file identifier of the tender document to which the substring contained in the similar substring pair belongs, matchStart represents the starting position of the substring in the corresponding tender document, and matchEnd represents the ending position of the substring in the corresponding tender document.

[0057] The MatchPair data structure allows for storing matching relationships in a smaller amount of storage space without storing the matched substring content, thus reducing memory usage.

[0058] In one embodiment of this application, the step of pairing and merging according to the pairing relationship to obtain the maximum repeating segment includes: Based on the pairing relationship, adjacent similar substring pairs are determined; Based on the preset data structure of the adjacent similar substring pairs, it is detected whether the substrings contained in the similar substring pairs meet the merging condition in both the document to be checked and the comparison document; If the merging conditions are met, adjacent similar substring pairs are merged into the largest repeating segment.

[0059] In this embodiment, substrings of different sizes are extracted using different window sizes. Based on the substrings of different sizes, corresponding pairing relationships and similar substring pairs are obtained. Since the pairing relationship includes the start and end positions of the substrings, adjacent similar substring pairs can be determined based on the start and end positions included in the pairing relationship. Then, based on the preset data structure of the adjacent similar substring pairs, it is detected whether the substrings contained in the similar substring pairs meet the merging conditions in both the document to be checked and the comparison document. The adjacent similar substring pairs that meet the merging conditions are merged into the maximum duplicate segment.

[0060] In the implementation, there may be overlapping content and / or situations where consecutive repeated content exceeds the maximum window size. After selecting one bidding document as the document to be checked for plagiarism, the remaining bidding documents are used as comparison documents. After retrieving all pairings related to the document to be checked, the pairings are sorted in ascending order by the `sourceStart` field to ensure that the order of the pairings is consistent with their position in the original document and that the pairings are adjacent. Subsequently, based on the `sourceStart`, `sourceEnd`, `matchStart`, and `matchEnd` fields of the `MatchPair` data structure for adjacent similar substring pairs, it is checked whether the substrings contained in adjacent similar substring pairs satisfy the condition that the adjacent substrings are consecutive or overlapping in both the document to be checked and the comparison documents. If the condition is met, they are merged into a larger repeated fragment; otherwise, the fragment is considered to be the largest consecutive repeated fragment and added to the list of largest repeated fragments. This step is repeated iteratively until the end of the original document is reached, ultimately obtaining all the largest repeated fragments.

[0061] For example, setting the maximum window size to 9 yields similar substring pairs for substring 1 ("Multiple professional and technical personnel:") and substring 2 ("Zhang Moumou, Wang Moumou, and Zhou"), meaning that substring 1 and substring 2 exist in both the document to be checked and the comparison document. Then, based on the MatchPair data structure of the similar substring pairs for substring 1 and substring 2, it is determined whether substring 1 ("Multiple professional and technical personnel:") and substring 2 ("Zhang Moumou, Wang Moumou, and Zhou") are consecutive in the document to be checked, and whether substring 1 ("Multiple professional and technical personnel:") and substring 2 ("Zhang Moumou, Wang Moumou, and Zhou") are consecutive in the comparison document. The system compares whether the substrings are consecutive in the document. If substring 1 ("Multiple professional and technical personnel:") and substring 2 ("Zhang Moumou, Wang Moumou and Zhou") are consecutive in both the document to be checked and the comparison document, then the larger repeating segment ("Multiple professional and technical personnel: Zhang Moumou, Wang Moumou and Zhou") is obtained. If the repeating segment ("Multiple professional and technical personnel: Zhang Moumou, Wang Moumou and Zhou") is the final repeating content, then it is taken as the largest repeating segment ("Multiple professional and technical personnel: Zhang Moumou, Wang Moumou and Zhou"). If there are similar substrings after "Multiple professional and technical personnel: Zhang Moumou, Wang Moumou and Zhou", then they can be merged to obtain the merged repeating segment, which is taken as the largest repeating segment.

[0062] This embodiment expands the length of the substring to find the longest consecutive adjacent substring, sorts the similar substrings by position, and then merges the adjacent substrings that are adjacent and consecutive in both the document to be checked and the comparison document, thereby obtaining the largest repeated segment of the most similar segment. This allows for the identification of more repeated segments and further overcomes the problem that existing text duplication detection technologies cannot detect the duplication rate of long professional terms.

[0063] In one embodiment of this application, the step of calculating the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document includes: The documents are merged based on the largest duplicate segment to obtain a merged document; Delete the preset template content from the merged document; Calculate the content duplication rate of the merged document after deleting the preset template content based on the maximum duplicate segment; The plagiarism detection result of the document to be checked is determined based on the content repetition rate.

[0064] In this embodiment, documents are merged based on the largest duplicate fragment to obtain a merged document, and preset template content is deleted from the merged document; then, the content duplication rate of the merged document after deleting the preset template content is calculated based on the largest duplicate fragment; finally, the duplication result of the document to be checked is determined based on the content duplication rate.

[0065] In practical implementation, since bidders all use the unified template and directory structure of the tendering party to fill in their tender documents, multiple tender documents will contain a large amount of duplicate pre-set template content. Therefore, it is not possible to simply accumulate duplicate fragments. In this embodiment, after merging documents based on the largest duplicate fragment, the pre-set template content is removed from the merged document. Then, the content duplication rate of the merged document after deleting the pre-set template content is calculated. The mathematical formula for calculating the content duplication rate is as follows:

[0066] in, This represents the length of the segment with the largest repeating segment. This represents the character length of the merged document obtained by merging documents based on the largest duplicate fragment, where n represents the character length of the preset template content, and |O| represents the text length of the original document to be checked for duplicates.

[0067] This application obtains the content of a bidding document, which includes a document to be checked for plagiarism and a comparison document. It extracts substrings from the document content, buckets these substrings, and maps substrings that meet preset conditions to the same hash bucket. It calculates the similarity between substrings in the document to be checked and substrings in the comparison document within the same hash bucket, and determines the pairing relationships between the substrings. Based on the pairing relationships, it merges the substrings to obtain the maximum duplicate segment. Finally, it calculates the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the bidding document. By using a bucketing strategy to group similar substrings into the same hash bucket, it achieves rapid duplication detection on large-scale text data, improving the response speed of plagiarism detection. Determining the maximum duplicate segment based on similarity and pairing relationships solves the problem of not being able to identify duplicate segments with only minor character modifications.

[0068] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0069] Reference Figure 2 The diagram illustrates a structural block diagram of a bid document plagiarism detection system provided in an embodiment of this application, which may specifically include the following modules: The tender document acquisition module 201 is used to acquire the file content of the tender document; the tender document includes the document to be checked for plagiarism and the comparison document. The substring extraction module 202 is used to extract substrings from the file content; Bucketing module 203 is used to perform bucketing processing on the substrings and map substrings that meet preset conditions to the same hash bucket; The pairing relationship determination module 204 is used to calculate the similarity between the substrings of the document to be deduplicated and the substrings of the comparison document in the same hash bucket, and to determine the pairing relationship between the substrings; The merging module 205 is used to perform pairing and merging according to the pairing relationship to obtain the maximum repeating segment; The plagiarism detection result generation module 206 calculates the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document.

[0070] In one embodiment of this application, the substring includes a first substring; the substring extraction module 202 includes: The first substring extraction submodule is used to slide the first window character by character in the file content according to the first window size, extract the string in each first window, until the entire file content is traversed to obtain the first substring; the first window size represents the size of the string that can be extracted from the first window.

[0071] In one embodiment of this application, the substring further includes a second substring; the system further includes: The second substring extraction submodule is used to expand the size of the first window, slide the second window character by character in the file content according to the second window size, and extract the string in each second window to obtain the second substring; the second window size represents the size of the string that can be extracted from the second window.

[0072] In one embodiment of this application, the binning module 203: The feature vector extraction submodule is used to extract the feature vector of the substring; The signature vector generation submodule is used to perform hashing processing on the obtained feature vector using the MinHash function to obtain the corresponding signature vector; The bucketing submodule is used to perform bucketing on the substrings according to the signature vector, and to map substrings whose signature vectors meet preset conditions to the same hash bucket.

[0073] In one embodiment of this application, the bucketing processing submodule includes: The first processing unit is used to divide the signature vector into a preset number of bands; The second processing unit is used to process the components of each band using a hash function to generate the corresponding hash bucket identifier. The third processing unit is used to map the substring corresponding to the signature vector to the same hash bucket if the values ​​of the components of the signature vector corresponding to the substring of the document to be checked and the signature vector corresponding to the substring of the comparison document meet the preset conditions in at least one band.

[0074] In one embodiment of this application, the pairing relationship determination module 204 includes: The deviation distance determination submodule is used to calculate the text character deviation distance between the substring of the document to be deduplicated and the substring of the comparison document in the same hash bucket; The similarity calculation submodule is used to calculate the similarity between the substring of the document to be checked and the substring of the comparison document based on the text character deviation distance; The pairing relationship determination submodule is used to store the pairing relationships between substrings whose similarity is greater than or equal to the similarity threshold according to a preset data structure, and to set the substrings whose similarity is greater than or equal to the similarity threshold as similar substring pairs.

[0075] In one embodiment of this application, the preset data structure includes: the file identifier of the tender document to which the substring in the similar substring pair belongs, the starting position of the substring in the similar substring pair, and the ending position of the substring in the similar substring pair.

[0076] In one embodiment of this application, the merging module 205 includes: The similar substring pair determination submodule is used to determine adjacent similar substring pairs based on the pairing relationship; The merge judgment submodule is used to detect whether the substrings contained in the adjacent similar substring pairs meet the merge condition in both the document to be checked and the comparison document, based on the preset data structure of the adjacent similar substring pairs. The merging submodule is used to merge adjacent similar substring pairs into the maximum repeating segment if the merging conditions are met.

[0077] In one embodiment of this application, the plagiarism detection result generation module 206 includes: The document merging submodule is used to merge documents based on the maximum duplicate segment to obtain a merged document. The template deletion submodule is used to delete preset template content in the merged document; The content duplication rate calculation submodule is used to calculate the content duplication rate of the merged document after deleting the preset template content based on the maximum duplicate segment. The plagiarism detection result generation submodule is used to determine the plagiarism detection result of the document to be checked based on the content repetition rate.

[0078] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0079] like Figure 3 As shown, in another embodiment provided in this application, an electronic device 300 is also provided, including a memory 310 and a processor 320. The memory 310 and the processor 320 are connected via a bus for communication. The memory 310 stores a computer program, which can run on the processor 320 to implement the above steps.

[0080] like Figure 4 As shown, in another embodiment provided in this application, a computer-readable storage medium 401 is also provided, which stores a computer program that implements the methods described in the above embodiments when executed by a processor.

[0081] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0082] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0087] Finally, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for detecting plagiarism in tender documents, characterized in that, The method includes: Obtain the content of the bidding documents; the bidding documents include the document to be checked for plagiarism and the comparison document. Extract substrings from the file content; The substrings are bucketed, and substrings that meet the preset conditions are mapped to the same hash bucket; Calculate the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and determine the pairing relationship between the substrings; Based on the pairing relationship, the pairs are merged to obtain the maximum repeating segment; The content duplication rate is calculated based on the maximum duplicate segment to determine the plagiarism detection result of the tender document.

2. The method according to claim 1, characterized in that, The substring includes a first substring, and the substring from which the file content is extracted includes: The first window is slid across the file content character by character according to its size, extracting the string within each window until the entire file content has been traversed, thus obtaining the first substring; the first window size represents the size of the string that can be extracted from the first window.

3. The method according to claim 2, characterized in that, The substring further includes a second substring. Before performing pairing and merging based on the pairing relationship to obtain the maximum repeating segment, the method further includes: The first window size is expanded, and the second window is slid through the file content character by character according to the second window size to extract the string in each second window, thus obtaining the second substring; the second window size represents the size of the string that can be extracted from the second window.

4. The method according to any one of claims 1-3, characterized in that, The step of bucketing the substrings and mapping substrings that meet preset conditions to the same hash bucket includes: Extract the feature vector of the substring; The obtained feature vector is hashed using the MinHash function to obtain the corresponding signature vector; The substrings are bucketed according to the signature vector, and substrings whose signature vectors meet preset conditions are mapped to the same hash bucket.

5. The method according to claim 4, characterized in that, The step of bucketing the substrings according to the signature vector, mapping substrings whose signature vectors meet preset conditions to the same hash bucket, includes: The signature vector is divided into a preset number of bands; A hash function is used to process the components of each band, generating a corresponding hash bucket identifier; If the signature vector corresponding to the substring of the document to be checked and the signature vector corresponding to the substring of the comparison document meet the preset conditions in at least one band, then the substring corresponding to the signature vector is mapped to the same hash bucket.

6. The method according to any one of claims 1-3, characterized in that, The calculation of the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and the determination of the pairing relationship between the substrings, includes: Calculate the text character deviation distance between the substring of the document to be checked and the substring of the comparison document in the same hash bucket; The similarity between the substrings of the document to be checked and the substrings of the comparison document is calculated based on the text character deviation distance. The pairing relationships between substrings with a similarity greater than or equal to the similarity threshold are stored according to a preset data structure, and the substrings with a similarity greater than or equal to the similarity threshold are set as similar substring pairs.

7. The method according to claim 6, characterized in that, The preset data structure includes: the file identifier of the tender document to which the substring in the similar substring pair belongs, the starting position of the substring in the similar substring pair, and the ending position of the substring in the similar substring pair.

8. The method according to any one of claims 1-3, characterized in that, The process of pairing and merging according to the pairing relationship to obtain the maximum repeating segment includes: Based on the pairing relationship, adjacent similar substring pairs are determined; Based on the preset data structure of the adjacent similar substring pairs, it is detected whether the substrings contained in the similar substring pairs meet the merging condition in both the document to be checked and the comparison document; If the merging conditions are met, adjacent similar substring pairs are merged into the largest repeating segment.

9. The method according to claim 1, characterized in that, The step of calculating the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document includes: The documents are merged based on the largest duplicate segment to obtain a merged document; Delete the preset template content from the merged document; Calculate the content duplication rate of the merged document after deleting the preset template content based on the maximum duplicate segment; The plagiarism detection result of the document to be checked is determined based on the content repetition rate.

10. A system for detecting plagiarism in tender documents, characterized in that, The system includes: The tender document acquisition module is used to acquire the file content of the tender documents; the tender documents include the document to be checked for plagiarism and the comparison document; The substring extraction module is used to extract substrings from the file content; The bucketing module is used to bucket the substrings and map substrings that meet preset conditions to the same hash bucket. The pairing relationship determination module is used to calculate the similarity between substrings of the document to be deduplicated and substrings of the comparison document in the same hash bucket, and to determine the pairing relationship between the substrings; The merging module is used to perform pairing and merging based on the pairing relationship to obtain the maximum repeating segment; The plagiarism detection result generation module calculates the content duplication rate based on the maximum duplicate segment to determine the plagiarism detection result of the tender document.

11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1-9.

Images