Bidding document duplicate checking and anomaly analysis method and system

Through Simhash improvement algorithm and multi-dimensional feature words to identify duplicate content in bid documents, combined with hash operation and exception database comparison, the problem of low efficiency and insufficient accuracy of bid document dilution checking and abnormal analysis in the existing technology is solved, and the problem of insufficient accuracy of bid documents is realized. Intelligent and scientific analysis.

CN118886409AInactive Publication Date: 2024-11-01SHANDONG SHENGLI PROJECT MANAGEMENT CO LTD

Patent Information

Application Number
CN202411375517.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art has problems such as low efficiency, insufficient accuracy, inability to identify non-text form abnormalities and lack of multi-dimensional analysis in bid document plagiarism checking and abnormal analysis, especially in the absence of whitelist filtering function and time positioning capabilities.

Method used

Simhash improved algorithm is used to calculate the text similarity of bid files, combine multi-dimensional feature words and hash operations to identify duplicate content, define an exception database and generate an exception report, identify duplicate images, and establish a local exception database for comparison.

Benefits of technology

It improves the accuracy and efficiency of bid document plagiarism checking and abnormal analysis, realizes intelligent, scientific and automated analysis, and can accurately locate abnormal content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118886409A_ABST
    Figure CN118886409A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of duplicate checking management, and particularly relates to a bidding document duplicate checking and exception analysis method and system, and the method comprises the steps: building an exception database containing a plurality of exception items; obtaining all bidding files of the same project or the same section; calculating the text similarity of any two bidding files based on a Simhash improved algorithm; identifying repeated contents in any two bidding files; the duplicated content comprises duplicated texts and duplicated pictures; defining repeated content conforming to any abnormal item in the abnormal database as abnormal data; and generating an exception report according to the exception data and the text similarity. According to the bidding document duplicate checking and anomaly analysis method, the accuracy and comprehensiveness of the bidding document duplicate checking and anomaly analysis are improved, the duplicate checking efficiency is improved through an intelligent analysis method, and the bidding document duplicate checking and anomaly analysis are intelligent, scientific and automatic.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of duplicate checking management, and in particular relates to a bidding document duplicate checking and abnormality analysis method and system. Background Art

[0002] With the improvement and standardization of the bidding market, most of the illegal behaviors have been effectively controlled, but there are still some bidding activities where the parties collude with each other, colluded in bidding, etc., which seriously disrupt the normal order of bidding activities and undermine fair competition. Therefore, how to quickly, efficiently and accurately check the duplicate and abnormal analysis of the bidding documents during the bid evaluation process has become an urgent problem to be solved.

[0003] Some existing duplicate checking and anomaly analysis methods have the following problems:

[0004] 1) Due to the different formats of bidding documents and different types and ranges of document content, problems such as low efficiency and insufficient accuracy of duplicate checking and exception analysis may occur.

[0005] 2) The lack of whitelist filtering function results in the identification of duplicate content that is too complex and cannot be accurately analyzed.

[0006] 3) Lack of multi-dimensional analysis, only text can be checked for plagiarism, and other forms of anomalies cannot be identified.

[0007] 4) Lack of analytical functions, unable to accurately locate abnormally consistent content during the bid evaluation process, resulting in poor actual use effect. Summary of the invention

[0008] In view of the defects in the prior art, the present invention provides a method and system for checking duplicate and analyzing anomalies in bidding documents, which improves the accuracy, comprehensiveness and efficiency of checking duplicate and analyzing anomalies in bidding documents.

[0009] In a first aspect, a method for checking duplicate and analyzing abnormalities in bidding documents comprises: Establish an abnormality database containing multiple abnormal items; Obtain all bidding documents for the same project or bidding section; Calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identify duplicate content between any two bidding documents; duplicate content includes duplicate text and duplicate images; Define the repeated content that meets any abnormal item in the abnormal database as abnormal data; Generate anomaly reports based on anomaly data and text similarity.

[0010] Furthermore, the text similarity of any two bidding documents is calculated based on the Simhash improved algorithm, specifically including: Perform word segmentation and filtering on the bidding documents to obtain multiple characteristic words; Convert feature words into space vectors; Merge and reduce the dimension of the space vectors respectively to obtain a merged vector; The text similarity of the two bidding documents is calculated using the merged vectors of the two bidding documents.

[0011] Furthermore, the word segmentation filtering of the bidding documents specifically includes: Convert the bidding documents into different formats; Segment the bidding document after format conversion according to punctuation marks or character string thresholds to obtain multiple keywords; Create a whitelist containing the tender information in the tender document; Delete the keywords in the whitelist to obtain the characteristic words.

[0012] Furthermore, the space vector is a multi-dimensional space vector; the unit direction of each dimension in the space vector represents a feature word, and the extension direction of each dimension represents the weight of the feature word; The weight is obtained by multiplying the first feature by the second feature; the first feature is obtained by dividing the number of times the feature word appears in the bidding document by the total number of feature words in the bidding document; the second feature is obtained by taking the logarithm of the quotient of the total number of bidding documents divided by the number of bidding documents containing the feature word.

[0013] Furthermore, using the merged vector of the two bidding documents to calculate the text similarity of the two bidding documents specifically includes: Perform hash operation on the combined vector of each bidding document respectively to obtain a hash value; Concatenate all hash values ​​in the bidding document to obtain a hash string; Calculate the Hamming distance between the hash strings in the two bid documents to get the text similarity.

[0014] Furthermore, the method for identifying duplicate images specifically includes: Extract the features of the image, including color features, edge features, and texture features; Calculate the Simhash value of each feature separately; Calculate the MD5 hash value of each image; When the MD5 hash values ​​of two images are the same, or when the MD5 hash values ​​of two images are different but the Hamming distance of the Simhash hash values ​​is less than the set threshold, the two images are considered similar.

[0015] Furthermore, the abnormal item includes a combination of one or more of the following:

[0016] 1) The time when the documents were created is the same;

[0017] 2) The document authors are the same;

[0018] 3) The last saver is the same;

[0019] 4) The last saved date is the same;

[0020] 5) The time of the last print is the same;

[0021] 6) The document creation date is the same;

[0022] 7) The document modification date is the same;

[0023] 8) The document access date is the same;

[0024] 9) The serial numbers of the devices are the same;

[0025] 10) The ID card and address of the legal representative or senior executives are the same;

[0026] 11) The ID numbers of the relevant persons are the same;

[0027] 12) The registered addresses of the bidding units are the same;

[0028] 13) The contact telephone numbers of the bidding units are the same;

[0029] 14) The risk warning information of the bidding units is the same;

[0030] 15) The contact person names of the bidding units are the same;

[0031] 16) The contact email addresses of the bidding units are the same;

[0032] 17) The names or abbreviations of other bidding units;

[0033] 18) The pictures are the same;

[0034] 19) The number of consecutive identical tab characters is greater than the set value; the tab characters do not exist in the bidding documents but exist in the tender documents;

[0035] 20) The text contains the same grammatical errors.

[0036] Furthermore, after obtaining the abnormal data, the method further includes: Establish a local anomaly database; Storing abnormal data of the bidding documents in an abnormality database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; Duplicate contents existing in the exception database are defined as exception data.

[0037] In a second aspect, a bidding document duplication checking and abnormality analysis system includes: Abnormal database: contains multiple abnormal items; Collection unit: used to obtain all bidding documents of the same project or the same bidding section; Similarity calculation unit: used to calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identification unit: used to identify repeated contents in any two bidding documents; repeated contents include repeated texts and repeated images; Abnormal identification unit: used to define the repeated content that meets any abnormal item in the abnormal database as abnormal data; Report unit: used to generate anomaly reports based on anomaly data and text similarity.

[0038] Furthermore, the abnormality identification unit is also used for: Establish a local anomaly database; Storing abnormal data of the bidding documents in an abnormality database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; Duplicate contents existing in the exception database are defined as exception data.

[0039] It can be seen from the above technical scheme that the bid document duplication checking and anomaly analysis method provided by the present invention improves the accuracy and comprehensiveness of bid document duplication checking and anomaly analysis, improves the efficiency of duplication checking through intelligent analysis methods, and makes bid document duplication checking and anomaly analysis intelligent, scientific and automated. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0041] Figure 1 A flowchart of a bidding document duplication checking and anomaly analysis method provided in an embodiment.

[0042] Figure 2 The following is a general flow chart of the text similarity calculation method provided in the embodiment.

[0043] Figure 3 A flowchart of a word segmentation filtering method provided in an embodiment.

[0044] Figure 4A flowchart of a method for calculating text similarity provided in an embodiment.

[0045] Figure 5 A flowchart of a repeated image recognition method provided in an embodiment.

[0046] Figure 6 A flowchart of an abnormal database screening method provided in an embodiment.

[0047] Figure 7 A module block diagram of the bidding document duplication checking and anomaly analysis system provided in the embodiment. DETAILED DESCRIPTION

[0048] The embodiments of the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and the protection scope of the present invention cannot be limited thereto. It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should be the common meanings understood by those skilled in the art to which the present invention belongs.

[0049] It should be understood that when used in this specification, the terms "include" and "comprising" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0050] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms.

[0051] As used in this specification, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0052] Example:

[0053] A method for checking duplicate bid documents and analyzing abnormalities, see Figure 1 ,include: Establish an abnormality database containing multiple abnormal items; Obtain all bidding documents for the same project or bidding section; Calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identify duplicate content between any two bidding documents; duplicate content includes duplicate text and duplicate images; Define the repeated content that meets any abnormal item in the abnormal database as abnormal data; Generate anomaly reports based on anomaly data and text similarity.

[0054] In this embodiment, the method for checking duplicates and analyzing abnormalities in bidding documents includes two steps: checking duplicates and analyzing abnormalities. When checking duplicates, the method mainly checks duplicates for all bidding documents of the same project or the same bidding section. The method obtains the repeated content of any two bidding documents by a pairwise comparison method, wherein the repeated content includes repeated text or repeated pictures. For example, the result of the duplicate check is: the repetition rate of Company A and Company B is 7.06%, the number of repeated words is 1702, and most of the repeated content appears in the maintenance safety and environmental protection measures in the technical part of the bidding documents, which are specifically reflected in the maintenance management system, standards and measurement management system, safety management measures, maintenance safety management measures, maintenance safety operating procedures, safety management system, management measures for civilized operations on site, environmental protection measures and other contents.

[0055] In this embodiment, after the method obtains the repeated content, it is also necessary to compare the repeated content with the abnormal items in the abnormal database to determine whether the repeated content meets the abnormal items. The abnormal items represent repeated content that should not appear. Then the method defines the repeated content that meets any abnormal item as abnormal data, and generates an abnormal report based on the abnormal data. For example, the abnormal report of bidding unit A shows: the repetition rate between bidding unit A and bidding unit B is 1.32%, and the number of repeated words is 388; the repetition rate between bidding unit A and bidding unit C is 1.69%, and the number of repeated words is 594; the repetition rate between bidding unit A and bidding unit D is 2.43%, and the number of repeated words is 948, and so on. The abnormal report can also count the location of all abnormal data in the bidding document.

[0056] The method for checking duplicate and analyzing anomalies in bidding documents improves the accuracy and comprehensiveness of checking duplicate and analyzing anomalies in bidding documents, and improves the efficiency of checking duplicate and analyzing anomalies through intelligent analysis methods, making checking duplicate and analyzing anomalies in bidding documents intelligent, scientific and automated.

[0057] Further, see Figure 2 , the text similarity of any two bidding documents is calculated based on the Simhash improved algorithm, specifically including: Perform word segmentation and filtering on the bidding documents to obtain multiple characteristic words; Convert feature words into space vectors; Merge and reduce the dimension of the space vectors respectively to obtain a merged vector; The text similarity of the two bidding documents is calculated using the merged vectors of the two bidding documents.

[0058] In this embodiment, the method performs word segmentation filtering on the bidding documents to obtain multiple feature words; converts the feature words into space vectors; merges and reduces the dimension of the space vectors respectively to obtain a merged vector; and uses the merged vector of the two bidding documents to calculate the text similarity of the two bidding documents. Among them, the text similarity is used to measure the similarity between the two bidding documents. For example, the cosine similarity of the two merged vectors can be calculated to determine the text similarity. Assuming that A and B are two n-dimensional space vectors, A is [A1, A2, ..., An], and B is [B1, B2, ..., Bn], then the cosine similarity of the angle θ between A and B is equal to:

[0059] The closer the cosine similarity is to 1, the closer the angle between the two merged vectors is to 0 degrees, that is, the more similar the two merged vectors are. The smaller the cosine similarity, the greater the text similarity; the larger the cosine similarity, the smaller the text similarity.

[0060] Further, see Figure 3 , the word segmentation filtering of the bidding documents specifically includes: Convert the bidding documents into different formats; Segment the bidding document after format conversion according to punctuation marks or character string thresholds to obtain multiple keywords; Create a whitelist containing the tender information in the tender document; Delete the keywords in the whitelist to obtain the characteristic words.

[0061] In this embodiment, the method first converts the format of the bidding document, wherein the format conversion includes several situations: 1) conversion between sub-formats of word documents; sub-formats include doc, docx, wps, etc. The method can use a word document converter to convert the format of the word document. The word document converter defines the Word Converter abstract class, and the format conversion of all word documents is implemented by inheriting the Word Converter class. 2) Format conversion of PDF documents. The method can use the PDFBox tool to convert the format of the PDF document. PDFBox is a Java class library using the BSD open source license. PDFBox can be used to read, create and edit PDF files. PDFBox is designed in an object-oriented way to describe PDF documents, and the data of the PDF document is regarded as a collection of a series of basic objects, and the basic objects are defined in the org.pdfbox.cos package. 3) Format conversion of Office documents. The method can use Apache POI to modify and read Office documents, which includes Office documents that convert user input data or files into other formats.

[0062] In this embodiment, after the format conversion is completed, the method can also perform word segmentation on the input bidding document, remove stop words and punctuation marks, and convert words to lowercase. Specifically, the method uses the following two methods to perform word segmentation: 1) using punctuation marks as word segmentation nodes for word segmentation; 2) performing word segmentation by a preset string threshold (i.e., a length threshold, for example, set to 7). The method can improve the accuracy of checking for duplicates in situations such as changing word order, punctuation, or paragraph order through word segmentation. The method can also perform word segmentation by semantics to obtain characters, words, phrases, etc. Finally, the method performs data filtering on the keywords obtained by word segmentation through a whitelist. For example, the method can also add some specified words or sentences in the bidding document to the whitelist. The method first creates a whitelist of relevant publicly released bidding information such as bidding documents. When the bidding document contains keywords that exist in the whitelist, it means that the keyword is a reasonable keyword that appears in the bidding document and does not need to be checked for duplicates. At this time, stop words can be used to replace the keywords that exist in the whitelist, and the remaining keywords are defined as feature words, thereby saving storage space and improving search efficiency.

[0063] For example, the bidding documents contain 572 words of "technical standards and requirements", which clearly stipulate the technical requirements, quality standards, and related problem handling. The bidding units quoted and made commitments without modification in their bidding documents. The repetition of these contents will not be counted as repeated contents.

[0064] Furthermore, the space vector is a multi-dimensional space vector; the unit direction of each dimension in the space vector represents a feature word, and the extension direction of each dimension represents the weight of the feature word; The weight is obtained by multiplying the first feature by the second feature; the first feature is obtained by dividing the number of times the feature word appears in the bidding document by the total number of feature words in the bidding document; the second feature is obtained by taking the logarithm of the quotient of the total number of bidding documents divided by the number of bidding documents containing the feature word.

[0065] In this embodiment, the method converts each feature word into a multidimensional space vector so that each feature word can be mapped to a numerical value to reflect its importance in the document. Since the collected feature words are all Chinese characters, they cannot be read by computers. Therefore, the method converts the feature words into space vectors that can be recognized by computers. The method can first map the feature words into space vectors, and then construct a space vector model of the bid document based on all the space vectors in the bid document. The unit direction of each dimension in the space vector represents the feature word, and the extension direction of each dimension represents the weight of the feature word. The larger the weight, the better the feature word can reflect the content of the bid document, and the smaller the weight, the feature word cannot be used.

[0066] In this embodiment, the method sets weights for each feature word to highlight important words and weaken unimportant words. Assume that the bidding document set P={p1,p2,…pn}, pn represents the nth bidding document; the feature word set T={t1,t2,…tn} of each bidding document, tn represents the nth feature word. The weight set Wi={wi1,wi2, ,…win}, win represents the weight of the nth space vector. Where Wij=tfij×idfk, tfij represents the frequency of the feature word tj in the bidding document, and idfk represents the inverse of the number of occurrences of the feature word tj in the bidding document. The method evaluates the importance of the feature word in the bidding document by counting the first feature and the second feature:

[0067] 1) The first feature: represents the frequency of the feature word appearing in the bidding document. The first feature = (the number of times the feature word appears in the bidding document) / (the total number of feature words in the bidding document).

[0068] 2) Second feature: indicates the general importance of the feature word. The fewer the bidding documents containing the feature word, the larger the second feature, indicating that the feature word has a good ability to distinguish categories. Second feature = log (total number of bidding documents) / (number of bidding documents containing the feature word).

[0069] 3) Weight: Weight = first feature × second feature. The second feature is usually smoothed by adding 1 to avoid the situation where the denominator is zero. This method can also normalize the weight. This method helps to solve the problem of not paying attention to words with high similarity in traditional keyword extraction methods and improve the accuracy of keyword extraction.

[0070] Further, see Figure 4 , using the merged vector of two bidding documents to calculate the text similarity of the two bidding documents specifically includes: Perform hash operation on the combined vector of each bidding document respectively to obtain a hash value; Concatenate all hash values ​​in the bidding document to obtain a hash string; Calculate the Hamming distance between the hash strings in the two bid documents to get the text similarity.

[0071] In this embodiment, the method can perform a hash operation on the merged vector of the bidding documents, such as binary hashing, to map a continuous numerical interval to 0 or 1. Then all hash values ​​are concatenated to form a long binary string, i.e., a hash string (Simhash value). Finally, the Hamming distance of the hash strings in the two bidding documents is calculated to obtain the text similarity. For example, if the Hamming distance is less than 3, it is considered that the text similarity of the two bidding documents is high and the two bidding documents are similar.

[0072] This method can also combine the full match algorithm and the fuzzy match algorithm to check for duplicate content. The full match algorithm can accurately identify exactly the same text content, ensuring that no plagiarism is missed by direct copying and pasting. However, the full match algorithm cannot accurately identify text that has been slightly modified (such as replacing synonyms, adjusting word order, etc.). The fuzzy match algorithm introduces natural language processing technology (NLP), such as stem extraction, word form restoration, and semantic similarity calculation, which can identify texts that are semantically similar but have different literal expressions. It can capture plagiarism that is missed by the full match algorithm and significantly improve the accuracy of checking for duplicate content.

[0073] The full match duplicate checking algorithm first splits the two bidding documents according to the fixed threshold length. By default, they are split according to their respective punctuation marks. The more commonly used 13 characters can also be used as the threshold length value. If the two bidding documents are completely identical, they are judged as duplicates. Currently, the more commonly used full match duplicate checking algorithms are the KMP algorithm and the AC algorithm.

[0074] The fuzzy matching duplicate checking algorithm first sorts the duplicate checking text and the text to be checked into sentences in lexicographic order, and sets a pointer for each sentence list of the two texts. If the lexicographic order of the sentence pointed to by the current pointer of the text to be checked is less than the lexicographic order of the text pointed to by the current pointer of the text to be checked, the pointer at the duplicate checking text is moved down by one, otherwise the pointer at the text to be checked is moved down by one. After each pointer movement, the cosine similarity of the two sentences is calculated, and it is determined whether it exceeds the threshold until the text to be checked or the duplicate checking text has been calculated.

[0075] Further, see Figure 5 , the identification method of repeated images specifically includes: Extract the features of the image, including color features, edge features, and texture features; Calculate the Simhash value of each feature separately; Calculate the MD5 hash value of each image; When the MD5 hash values ​​of two images are the same, or when the MD5 hash values ​​of two images are different but the Hamming distance of the Simhash hash values ​​is less than the set threshold, the two images are considered similar.

[0076] In this embodiment, the method combines the Simhash algorithm with the MD5 algorithm to determine whether two pictures are similar. MD5 is mainly used to generate hash values ​​of data, while Simhash is a local sensitive hash algorithm. The method first preprocesses the picture and converts it into a format that is more suitable for Simhash processing, including first extracting useful features from the picture, including color features, edge features, and texture features. The method can also consider using more complex image recognition technologies, such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Robust Features) algorithms to extract image features. The method then quantizes the extracted features into binary vectors or digital vectors for Simhash processing, and the quantization method can be implemented by threshold segmentation, hash functions, and other methods. Then the method applies the Simhash algorithm to generate the Simhash hash value of the picture and calculates the MD5 hash value of each picture. The MD5 hash value can be used as a unique identifier for the picture to quickly filter out exactly the same pictures. Finally, in the process of checking for duplicates, the MD5 hash value is first used to quickly find exactly the same pictures. For pictures with different MD5 values, the Simhash hash value is used to find pictures that are visually similar but not exactly the same in content. When comparing the Simhash values ​​of images, the Hamming distance can be used to calculate the similarity between the Simhash values. This method can also set a similarity threshold. When the similarity of two Simhash values ​​exceeds the similarity threshold (such as 80%), the two images are considered similar.

[0077] For example, suppose there are two images A and B. Their color histograms are simplified as follows: the color histogram of image A: red

[50] , green

[100] , blue

[150] ; the color histogram of image B: red

[45] , green

[105] , blue

[145] . Then the method maps each color interval of the color histogram to a binary bit and sets the threshold to 75. Then the features of image A are simplified to: 101 (red>75, green<75, blue>75), and the features of image B are simplified to: 101 (red>75, green<75, blue>75). The Simhash hash values ​​of images A and B are 10101 and 10110 respectively. The MD5 hash value of image A is: abcdef1234567890; the MD5 hash value of image B is: bbcdef1234567891. When checking for duplicates, it is found through the MD5 hash value that images A and B are not exactly the same. The Hamming distance between the two Simhash hash values ​​is calculated to be 1. Assuming the threshold is 3, since the Hamming distance is less than the threshold, it is determined that images A and B are similar. This method combines Simhash with MD5 to determine duplicate images. While maintaining the fast and compact characteristics of the MD5 algorithm, it can use the efficiency of the Simhash algorithm to improve the performance of document duplicate checking or similarity detection. At the same time, by improving the Simhash algorithm, the accuracy of recognition can be further improved.

[0078] The following example illustrates how to calculate the text similarity of two bidding documents: Step 1: Text Preprocessing Perform word segmentation on the two sentences: - Sentence 1 (d1): "In order to ensure the reliable operation of the valve, the valve must be regularly maintained by the technicians of Company A or personnel trained by Company A." - Sentence 2 (d2): "Regular maintenance of the valve is essential and should be performed by professional technicians from Company A to ensure its long-term stable operation."

[0079] Step 2: Calculate weights Calculate the weight of each word in the sentence. Assuming there are only these two sentences in the bidding document, First feature = (number of times feature word t appears in bid document d) / (total number of feature words in bid document d); Second feature = log(total number of bidding documents / number of bidding documents containing feature word t) For example, if "valve" appears 2 times in sentence 1 and 1 time in sentence 2, then its first feature in sentence 1 is 2 / 8 and its first feature in sentence 2 is 1 / 5. Since the total number of documents is 2, the second feature is log(2 / 1).

[0080] Step 3: Simhash signature generation The Simhash algorithm is used to convert the weight of each word into a Simhash signature.

[0081] Step 4: Cosine similarity calculation Calculate the cosine similarity between two bid documents.

[0082] Step 5: Check for plagiarism If the cosine similarity of two documents is greater than a certain threshold (for example, 0.8), the two documents are considered similar and may need further duplication checking.

[0083] For example, suppose the combined vector of sentence 1 and sentence 2 is: - A = [0.3, 0.4, 0.1, 0.2] - B = [0.2, 0.3, 0.4, 0.1] =0.8 The cosine similarity of 0.8 is equal to the threshold of 0.8, so sentences 1 and 2 are similar.

[0085] Furthermore, in some embodiments, the abnormal item includes a combination of one or more of the following:

[0086] 1) The time when the documents were created is the same;

[0087] 2) The document authors are the same;

[0088] 3) The last saver is the same;

[0089] 4) The last saved date is the same;

[0090] 5) The time of the last print is the same;

[0091] 6) The document creation date is the same;

[0092] 7) The document modification date is the same;

[0093] 8) The document access date is the same;

[0094] 9) The serial numbers of the devices are the same;

[0095] 10) The ID card and address of the legal representative or senior executives are the same;

[0096] 11) The ID numbers of the relevant persons are the same;

[0097] 12) The registered addresses of the bidding units are the same;

[0098] 13) The contact telephone numbers of the bidding units are the same;

[0099] 14) The risk warning information of the bidding units is the same;

[0100] 15) The contact person names of the bidding units are the same;

[0101] 16) The contact email addresses of the bidding units are the same;

[0102] 17) The names or abbreviations of other bidding units;

[0103] 18) The pictures are the same;

[0104] 19) The number of consecutive identical tab characters is greater than the set value; the tab characters do not exist in the bidding documents but exist in the tender documents;

[0105] 20) The text contains the same grammatical errors.

[0106] In this embodiment, the anomaly detection of the method includes the following 20 items:

[0107] 1) Check if the creation time of the documents of the two bidding documents is the same. Since different bidding documents are made by different people, the document creation date should be different. Therefore, when the document creation time of the two bidding documents is the same, it is determined that the document creation time of the two bidding documents is abnormally consistent. When performing the above time extraction, the 8-digit number corresponding to the date in the document attributes can be extracted for comparison. If they are the same, then compare the 6-digit number of the time and display the comparison result.

[0108] Using the PropertySystemView software function to view the system file properties, you can display various properties of the file, including but not limited to creation time, modification time, access time, author, title, comments, tags, etc. Through the PropertySystemView interface or command line function, load two bidding files that need to be detected (assuming they are file A and file B). Read and display various property information of each file, including the file name and creation time of the document. The creation time of the document is usually stored in the metadata of the file, which is recorded by the file system when the file is first created. PropertySystemView obtains this information by accessing the API of the file system.

[0109] Then compare the document creation time of file A and file B. First, compare the 8-digit date; if they are the same, then compare the 6-digit time. If the two times are exactly the same, it can be considered that the document creation time of the two bidding documents is the same. If the two bidding documents are created at the same time, it is very likely that they are copied from the same document, and there is a high possibility of abnormal consistency.

[0110] 2) Check whether the document authors of the two bidding documents are the same. Since different bidding documents are produced by different people, the authors should be different. Therefore, when the authors of the two bidding documents are the same, it is necessary to determine the type of situation in which the authors of the two bidding documents are abnormally consistent.

[0111] First, exclude common characters such as admin, administrator, user, lenovo, and website names. Then, exclude the situation where the bidder uses the template in the bidding document to directly edit and modify it into the bidding document. Finally, it is also possible that different people set the same nickname or there are people with the same name in the Office document by chance, but this requires a reasonable explanation. If it does not fall into the above situation, it is very likely that the bidding documents of different bidders were modified and edited by the same person, and there is a high possibility of abnormal consistency.

[0112] 3) Check whether the last savers of the two bidding documents are the same. Since different bidding documents are prepared by different personnel, the last savers should be different. Therefore, when the last savers of the two bidding documents are the same, it can be determined item by item based on the above circumstances that the last savers of the two bidding documents are abnormally consistent.

[0113] 4) Check whether the last save time of the two bidding documents is the same. Since different bidding documents are made by different people, the last save time should be different. Therefore, when the last save time of the two bidding documents is the same, it can be directly determined that the last save time of the two bidding documents is abnormally consistent. In view of the time delay of the save operation, we need to moderately relax the accuracy of the last save time check. The specific strategy is: as long as the two save operations occur within the same hour of the same day, it can be regarded as a potential abnormal consistency, and corresponding processing or verification can be performed.

[0114] 5) Check whether the last printing time of the two bidding documents is the same. Since different bidding documents are made by different people, the last printing time of the documents is different. Therefore, when the last printing time of the two bidding documents is the same, it is determined that the last printing time of the two bidding documents is abnormally consistent. When performing the above time extraction, the 8-digit number corresponding to the date in the document attributes can be extracted for comparison. If they are the same, then compare the 6-digit number of the time and display the comparison result.

[0115] First, if the last printing time of two or more bidding documents is highly consistent, especially when these documents have significant similarities in content and format, this may indicate suspicion of document duplication or tampering.

[0116] Secondly, if the bidding documents are printed very close to the bidding deadline, it may indicate that the bidders were not fully prepared due to time constraints, or that the documents were put together on the spur of the moment. This information is of great reference value to the tenderer and helps to identify and respond to possible risks in advance.

[0117] Finally, the printing time can be used as an auxiliary means to verify the source of the bidding documents. For example, if there is an obvious contradiction between the printing time of a bidding document and the start time of the bidding work, this may arouse further attention from the tenderer and require them to take other measures to verify the authenticity of the document.

[0118] Therefore, the same last printing time can be used as a criterion for determining abnormal consistency.

[0119] 6) Check the document creation dates of the two bidding documents to confirm whether they are the same. Given that different bidding documents are usually independently produced by different personnel, theoretically there should be differences in their creation times. Therefore, once it is found that the document creation dates of the two bidding documents are exactly the same, it can be determined that there is an abnormal consistency between the two bidding documents.

[0120] 7) Check whether the document modification dates of the two bidding documents are the same. Bidding documents are usually prepared independently by different personnel, and it is expected that there will be differences in their modification dates. Therefore, once it is found that the document modification dates of the two bidding documents are exactly the same, this is regarded as a signal of abnormal consistency. The 8-digit format of the modification date (usually including year, month and day) is accurately extracted from the attribute information of the document, and a preliminary comparison is first performed. If this part of the date is completely consistent, it is further refined to compare the 6-digit time (including hours, minutes, etc.), and the final comparison results are clearly presented to analyze whether there are any abnormalities.

[0121] 8) Check whether the document access dates of the two bidding documents are the same. Since different bidding documents are produced by different personnel, the document access time should be different in theory. Therefore, it is found that the document access time of the two bidding documents is exactly the same, and it is determined that the document access time of the two bidding documents is abnormally consistent. Extract the 8-digit representation of the access date (usually including year, month, and day) from the attributes of the document, and first perform a preliminary comparison. If the 8-digit digits of the two dates are exactly the same, further drill down to the 6-digit digits of the time (such as hours and minutes) for comparison, and clearly display the comparison results to analyze whether there are any abnormalities.

[0122] 9) Check whether the equipment serial numbers of the two bidding documents are the same. Under normal circumstances, the equipment serial numbers in different bidding documents should be independent and different. Therefore, when the same equipment serial numbers appear in two bidding documents, it is determined that the two bidding documents are abnormally consistent. When extracting the equipment serial number, extract 6 characters for comparison. Accurately extract the 6 characters of the equipment serial number from the bidding document (or adjust according to the actual serial number length), and then compare them one by one. If the comparison result shows that the two serial numbers are exactly the same, it can be determined that the equipment serial numbers are abnormally consistent.

[0123] 10) Check whether the ID card addresses of the legal representatives or senior executives of the two bidding documents are the same. Generally, since the legal representatives or senior executives of different bidding units have independent identities, their ID card addresses should also be different. Therefore, when the ID card address information of the legal representatives or senior executives in the two bidding documents is exactly the same, it is determined that the two bidding documents are abnormally consistent, which may mean that there is some unusual connection between the two units.

[0124] In order to accurately compare this information, the following steps will be taken: first, extract the ID card image from the bidding document; then, use professional text recognition technology (OCR) to process the image and accurately extract the address information, which is usually included in the address column of the ID card; finally, compare the extracted address information in detail to determine whether it is completely consistent. If the comparison results show that the address information is the same, it can be determined that the two bidding documents are abnormally consistent.

[0125] 11) Check whether the ID numbers of the relevant personnel in the two bidding documents are the same. The relevant personnel include members of the project management organization. According to conventional logic, different bidding documents represent different bidding units, and these units should arrange different members to participate when establishing project management organizations, so their ID numbers should be unique.

[0126] However, if the ID numbers of the project management organization members listed in the two bidding documents are exactly the same, this is a significant abnormal signal, indicating that the same person appears in the two bidding documents. The ID images of the relevant personnel can be extracted from key documents such as the power of attorney and the bidding letter. Subsequently, these images are accurately processed using advanced text recognition technology (OCR) to extract the ID numbers. Finally, these extracted ID numbers are compared one by one. Once the ID numbers of the relevant personnel in the two bidding documents are the same, it can be confirmed that there is an abnormal consistency.

[0127] 12) Check whether the registered addresses of the bidders in the two bid documents are the same. Since each bidder has its own unique legal entity identity, their registered addresses should also be independent of each other to reflect their respective business premises and legal addresses. In order to accurately compare the registered addresses, relevant information will be directly extracted from the basic information documents or business license images submitted by the bidders. These documents and images, as proof of official documents, can provide accurate and authoritative registered address data.

[0128] Then make a detailed comparison of the registered addresses in the two bidding documents. Pay attention to check every part of the address, including street name, house number, city, state / province and zip code, etc. Normally, the registered addresses of different bidding units are different. So when the registered addresses of the bidding units in the two bidding documents are the same, it is determined that the addresses of the two bidding documents are abnormally consistent.

[0129] 13) Check whether the contact numbers of the bidders in the two bid documents are the same. In conventional business practices, different bidders usually use different contact numbers to maintain their independence and smooth communication. So when the contact numbers of two bid documents are the same, the contact numbers of the two bid documents are judged to be abnormally consistent. This may mean that there is some undisclosed relationship between the bidders. In order to complete this test efficiently and accurately, the public information on third-party websites will be used for query. Through automated tools or manual verification, the contact numbers listed by each bidder will be extracted from the third-party website, and it will be ensured that these numbers are in a complete 11-digit format to meet the conventional telephone number specifications. Subsequently, the two extracted contact numbers are strictly compared. If the comparison result shows that the numbers are exactly the same, the contact numbers of the two bid documents are judged to be abnormally consistent.

[0130] 14) Check whether the risk warnings of the bidders in the two bid documents are the same. In a normal bidding environment, due to the different market environments, financial conditions, operating strategies and other factors faced by each bidder, the risk warning content should show unique differences. Therefore, if it is found that the risk warning content of the two bidders is exactly the same, this will be regarded as a significant anomaly, which may mean that there is some unspoken connection or information duplication between the bidders.

[0131] Use the public resources of the official website to query and obtain the risk warning information independently released by each bidding unit. Through careful comparison and analysis, when the risk warnings of two bidding documents are the same, it is determined that the risk warnings of the two bidding documents are abnormally consistent.

[0132] 15) Check whether the contact names of the two bidding documents are the same. In the normal bidding process, since each bidding unit is an independent business entity, different contacts are usually designated to handle various matters related to the bidding, so the contact names should be unique and different. After excluding the same name and surname, when the contact names of the two bidding documents are the same, it is determined that the contact names of the two bidding documents are abnormally consistent. The contact name can be extracted from the basic information of the bidding unit.

[0133] 16) Check whether the contact email addresses of the two bidding documents are the same. In normal commercial bidding practices, each bidding unit usually assigns a unique email address to its contact person in order to maintain the independence of communication and the security of information. Therefore, under normal circumstances, the contact email addresses of different bidding units should be different. Therefore, when the contact email addresses of the two bidding documents are the same, it is determined that the contact email addresses of the two bidding documents are abnormally consistent. The contact email address can be extracted from the basic information of the bidding unit.

[0134] 17) Check whether the names or abbreviations of the bidders in the two bid documents are the same. As a rule, each bidder has a unique name and possible official abbreviation to identify its independent business identity. The exact same names or abbreviations of the bidders in the two bid documents will be considered an abnormal situation. This situation may indicate that there is a certain relationship between the bidders or there is duplication of text.

[0135] Using the basic principles and operations of the previous bidding document duplicate checking, the full names and official abbreviations of all bidding units are filled in the query text box, and the names and abbreviations of the bidding units in the two bidding documents are extracted and compared through the specified query function. Therefore, when the names or abbreviations of the bidding units in the two bidding documents are the same, it can be determined that the two bidding documents are abnormally consistent.

[0136] 18) Check whether the images in the two bidding documents are the same. Usually, each bidding unit will prepare exclusive image materials according to its unique needs such as project proposals, product displays or company introductions. Therefore, the sources and contents of images in different bidding documents should be different to reflect their uniqueness and originality. If the same image is used in two bidding documents, the images will appear abnormally consistent. This situation may indicate that there is sharing, theft or improper copying of image materials between bidding units.

[0137] In order to accurately identify whether the images are the same, we will use the image recognition technology described earlier in this article to compare the images in the two bidding documents one by one. It can quickly identify subtle differences in key features, color distribution, pixel patterns, etc. in the images, so as to accurately determine whether the images are the same.

[0138] In order to detect whether the pictures are the same, we can use a variety of technical means, including but not limited to:

[0139] ① Image hash value comparison: Convert each image into a hash value (such as MD5), and then compare the hash values ​​of the images in the two bidding documents to see if they are consistent. If the hash values ​​are the same, it is highly suspected that the two images are exactly the same.

[0140] ② Visual similarity detection: For some images that may have been slightly modified (such as cropping, rotation, adjusting brightness / contrast, etc.) but still keep the basic content unchanged, visual similarity detection algorithms can be used to identify the similarities between them. These algorithms can analyze the color, texture, shape and other features of the image and give a similarity score.

[0141] ③ Metadata check: Image files usually contain rich metadata, such as shooting time, camera model, geographic location, etc. By checking these metadata, we can obtain additional information about the source of the image, which helps determine whether the images may come from the same source.

[0142] ④ Manual review: In some cases, especially when technical detection methods cannot give a clear conclusion, professionals can be arranged to conduct manual review. Manual review can analyze the content of the image more carefully and identify subtle differences or similarities.

[0143] Once it is confirmed that the same image is used in two bidding documents, this will be considered a case of abnormal image consistency.

[0144] 19) Check whether the number of consecutive identical tabs in two bidding documents is greater than a set value, where the tab does not exist in the bidding document but exists in the bidding document. Since different bidding documents are usually independently produced by different personnel or teams, in theory, there should be significant differences in the tab usage patterns between them. Therefore, when the number of identical tabs in two bidding documents is greater than a set value, it is determined that the two bidding documents have a large number of identical tabs.

[0145] The process of detecting whether the number of consecutive identical tabs in two bidding documents is greater than the set value is divided into: identification of tabs, determination of continuity, number statistics, and final comparison with the set value. The following is an example of the detection process:

[0146] ① Identification of tab characters: First, it is necessary to clarify how tab characters are represented in text files, usually \t in ASCII code or U+0009 in Unicode. All tab character sequences are extracted from the two bidding documents. These sequences may consist of a single tab character or multiple consecutive tab characters.

[0147] ② Detection of consecutive tabs: Traverse the text content of the bidding document, identify and mark all tab positions. Check whether tabs appear consecutively, that is, one tab is followed by another, and no other characters (such as spaces, line breaks, etc.) are inserted. Record the length of each consecutive tab sequence.

[0148] ③Quantity statistics and comparison: For two bidding documents, the maximum length of the continuous tab sequence in each file is counted. If there are continuous tab sequences with a length exceeding the set value (such as 20) in both files, and the position or content of these sequences in the files (if the text following the tab is also the same) is also highly similar, further analyze the context in which these tabs appear.

[0149] ④ Contextual analysis: Analyze the text content before and after the consecutive tabs to see if there are other similarities, such as the same format, vocabulary, sentence structure, etc. If other significant similarities are found in addition to the consecutive tabs, then this increases the possibility that the two bid documents may have been produced by the same or similar sources.

[0150] ⑤ Determination and subsequent processing: If, through the above analysis, it is determined that there are a large number of consistent continuous tab characters in the two bidding documents, and this consistency is not accidental or caused by a common template, then it can be determined that the two bidding documents have abnormal consistency in the use of tab characters.

[0151] Considering that different bidding units may use the same template or tools to prepare bidding documents, in-depth contextual analysis of the detected similarities is required to determine whether they truly constitute anomalies.

[0152] 20) Check whether the text and grammatical errors of the two bidding documents are the same. Under normal circumstances, different bidding documents are produced by different teams, so the text and grammatical errors in them should be different. If the text and grammatical errors in the two bidding documents are unusually consistent, this may indicate that there is some kind of correlation between them. In order to detect this consistency, the following methods and steps can be used:

[0153] ① Choose a suitable grammar checker. Third-party grammar checker: Choose a reliable and widely recognized grammar checker. These tools usually have the ability to check the grammar and spelling of words, punctuation, proper nouns, numerical values, etc. Intelligent cloud text service: Consider using cloud-based natural language processing services, which usually provide higher-level grammar analysis and error detection functions and can process large-scale text data.

[0154] ② Preprocess the bidding documents. Format conversion: Ensure that the bidding documents are in a processable text format, such as TXT, DOCX or PDF (need to be converted to text). Text cleaning: Remove non-text content in the document, such as pictures, tables, headers and footers, etc., and only keep the plain text part. Segment processing: Divide the document into manageable paragraphs or chapters for separate analysis.

[0155] ③ Use grammar checking tools for analysis. Analyze separately: Check the grammar of the two bidding documents separately, and record all grammatical errors and spelling errors in each document. Error classification: Classify errors by type, such as punctuation errors, improper vocabulary usage, sentence structure errors, etc.

[0156] ④ Compare the grammatical errors of the two files. Error list comparison: Compare the grammatical error lists of the two files to find the same errors. Error location analysis: Not only compare the types of errors, but also analyze whether the specific locations of the errors in the files are similar or identical. For example, the same grammatical errors in two files appear in similar sentence structures or paragraphs. Error context: Consider the text content around the error to see if there are other similarities, such as the same expression or logical structure.

[0157] ⑤ Determination and subsequent processing. Abnormal consistency determination: If there are multiple identical grammatical errors in the two documents, and the positions and contexts of these errors in the documents are also similar, it can be determined that the two bidding documents are abnormally consistent in terms of textual grammatical errors. Further investigation: According to the bidding regulations and procedural requirements, further investigation is conducted on possible abnormal situations, including requiring the bidding unit to explain the reasons and provide detailed information on the document production process. Report and record: The test results and investigation process are recorded for subsequent reference or use as evidence.

[0158] Therefore, when the grammatical errors in the text of two bidding documents are the same, it is determined that the grammatical errors in the text of the two bidding documents are extremely consistent.

[0159] Further, see Figure 6 , after getting the abnormal data, it also includes: Establish a local anomaly database; Storing abnormal data of the bidding documents in an abnormality database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; Duplicate contents existing in the exception database are defined as exception data.

[0160] In this embodiment, the method can also store abnormal data of the bidding document in the local database. When the bidding document of the bidding unit is received again, the repeated content of the bidding document is first compared with the abnormal data stored locally to determine whether the bidding document still contains the same abnormal data as last time. After eliminating the abnormal data that is the same as last time, the repeated content is compared with the abnormal items.

[0161] A bidding document duplication checking and abnormality analysis system, see Figure 7 ,include: Abnormal database: contains multiple abnormal items; Collection unit: used to obtain all bidding documents of the same project or the same bidding section; Similarity calculation unit: used to calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identification unit: used to identify repeated contents in any two bidding documents; repeated contents include repeated texts and repeated images; Abnormal identification unit: used to define the repeated content that meets any abnormal item in the abnormal database as abnormal data; Report unit: used to generate anomaly reports based on anomaly data and text similarity.

[0162] Furthermore, the abnormality identification unit is also used for: Establish a local anomaly database; Storing abnormal data of the bidding documents in an abnormality database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; Duplicate contents existing in the exception database are defined as exception data.

[0163] The system provided in the embodiment of the present invention is briefly described. For matters not mentioned in the embodiment part, reference may be made to the corresponding contents in the aforementioned embodiment.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for checking duplicate and analyzing abnormalities in bidding documents, characterized in that: include: Establish an abnormality database containing multiple abnormal items; Obtain all bidding documents for the same project or bidding section; Calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identify duplicate content in any two bidding documents; the duplicate content includes duplicate text and duplicate images; Define the repeated content that meets any abnormal item in the abnormal database as abnormal data; An exception report is generated based on the exception data and the text similarity.

2. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 1 is characterized in that: The text similarity calculation of any two bidding documents based on the Simhash improved algorithm specifically includes: Performing word segmentation filtering on the bidding document to obtain a plurality of characteristic words; Convert the feature words into space vectors; Merging and reducing the dimensions of the space vectors respectively to obtain a merged vector; The text similarity of the two bidding documents is calculated using the merged vectors of the two bidding documents.

3. The method for checking duplicate bid documents and analyzing abnormalities according to claim 2 is characterized in that: The word segmentation filtering of the bidding document specifically includes: Convert the format of the bidding document; Segment the bidding document after format conversion according to punctuation marks or character string thresholds to obtain multiple keywords; Create a whitelist containing the tender information in the tender document; The keywords existing in the white list are deleted to obtain the characteristic words.

4. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 2 is characterized in that: The space vector is a multi-dimensional space vector; the unit direction of each dimension in the space vector represents the feature word, and the extension direction of each dimension represents the weight of the feature word; The weight is obtained by multiplying the first feature by the second feature; The first feature is obtained by dividing the number of times the feature word appears in the bidding document by the total number of feature words in the bidding document; The second feature is obtained by taking the logarithm of the quotient of the total number of bid documents divided by the number of bid documents containing the feature word.

5. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 2 is characterized in that: Calculating the text similarity of two bidding documents using the merged vector of the two bidding documents specifically includes: Perform hash operation on the combined vector of each bidding document respectively to obtain a hash value; Concatenate all hash values ​​in the bidding document to obtain a hash string; The Hamming distance between the hash strings in the two bidding documents is calculated to obtain the text similarity.

6. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 1 is characterized in that: The method for identifying duplicate images specifically includes: Extracting features of the image, wherein the features include color features, edge features, and texture features; Calculate the Simhash value of each feature separately; Calculate the MD5 hash value of each image; When the MD5 hash values ​​of two pictures are the same, or when the MD5 hash values ​​of two pictures are different but the Hamming distance of the Simhash hash values ​​is less than a set threshold, the two pictures are determined to be similar.

7. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 1 is characterized in that: The abnormal items include one or more of the following combinations: 1) The time when the documents were created is the same; 2) The document authors are the same; 3) The last saver is the same; 4) The last saved date is the same; 5) The time of the last print is the same; 6) The document creation date is the same; 7) The document modification date is the same; 8) The document access date is the same; 9) The serial numbers of the devices are the same; 10) The ID card and address of the legal representative or senior executives are the same; 11) The ID numbers of the relevant persons are the same; 12) The registered addresses of the bidding units are the same; 13) The contact telephone numbers of the bidding units are the same; 14) The risk warning information of the bidding units is the same; 15) The contact person names of the bidding units are the same; 16) The contact email addresses of the bidding units are the same; 17) The names or abbreviations of other bidding units; 18) The pictures are the same; 19) The number of consecutive identical tab characters is greater than the set value; the tab characters do not exist in the bidding documents but exist in the tender documents; 20) The text contains the same grammatical errors.

8. The method for checking duplicate and analyzing abnormalities in bidding documents according to claim 1, characterized in that: After getting the abnormal data, it also includes: Establish a local anomaly database; storing the abnormal data of the bidding document in the abnormal database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; The repeated contents existing in the abnormal database are defined as abnormal data.

9. A bidding document duplication checking and abnormality analysis system, characterized in that: include: Abnormal database: contains multiple abnormal items; Collection unit: used to obtain all bidding documents of the same project or the same bidding section; Similarity calculation unit: used to calculate the text similarity of any two bidding documents based on the Simhash improved algorithm; Identification unit: used to identify repeated contents in any two bidding documents; the repeated contents include repeated texts and repeated images; Abnormal identification unit: used to define the repeated content that meets any abnormal item in the abnormal database as abnormal data; Report unit: used for generating an exception report according to the exception data and text similarity.

10. The bidding document duplication checking and abnormality analysis system according to claim 9 is characterized in that: The abnormality identification unit is also used for: Establish a local anomaly database; storing the abnormal data of the bidding document in the abnormal database; When the duplicate content of the bidding document is received again, the duplicate content is compared with the abnormal data of the corresponding bidding document in the abnormal database; The repeated contents existing in the abnormal database are defined as abnormal data.

Citation Information

Patent Citations

  • Detection method and system for bid enclosing and bid stringing and storage medium

    CN112819565A

  • Bidding behavior detection method and device based on electronic bidding file comparison

    CN115249007A

  • Intelligent fraud website identification method based on computing power network

    CN116996269A

  • Intelligent extraction method and system for key knowledge points of document based on docx format

    CN117764051A

  • Rich text content duplicate checking method, processor and storage medium

    CN118643819A

Cited By

  • Text duplicate checking and highlighting method and device, storage medium and program product

    CN119272721A

  • Project information analysis and duplicate checking method

    CN120162301A