Intelligent extraction method and system for key knowledge points based on docx format document

Abstract

The application provides a kind of based on docx format document key knowledge point intelligent extraction method and system, comprising: step S1: extracting tender material information, eliminating the reference to tender material;Step S2: a pair of bid file A file and B file are taken from combination randomly, with B document and tender material as reference object, the A document is checked for duplication;Step S3: the paragraph text is checked for duplication;Step S4: the table, picture, document attribute is checked for duplication.The application fully considers the characteristics of bidding activities, and eliminates the reasonable reference part of the tender material in the duplicate content of the bid, so that the statistical result is more accurate.And in the test result, also consider the picture object and the document attribute object, compared with the pure text duplication checking technology, it is more comprehensive.

Description

Technical Field

[0001] This invention relates to the field of computer plagiarism detection technology, specifically to a method and system for intelligent extraction of key knowledge points from docx format documents. Background Technology

[0002] The technical background of this invention mainly focuses on the field of XML markup language and text plagiarism detection. The Microsoft-defined Word XML standard is a structured markup language used to describe the content, format, and layout of Word documents. A docx document can be viewed as a file package with its root node path " / ". Under the root node, there are typically files such as "[Content_Type].xml", the word folder, the docPros folder, and _rels, which can be accessed through the root path. All XML files can be viewed as a "tree" object or a "sub-package" object. [Content_Type].xml stores predefined XML file types, which specify how to handle various tree objects. The _rels file specifies the relationships between tree objects. The main information of the document in the docx file is stored in document.xml. The plagiarism detection algorithms in this invention directly retrieve target nodes from document.xml or obtain target nodes from other trees through the relationships between trees. Different plagiarism detection strategies are designed according to different node types.

[0003] On the other hand, there are currently three main forms of text deduplication calculation. First, hash-based deduplication technology maps text to a fixed-length hash value using a hash function, and then compares whether the hash values ​​are the same to determine their similarity. The advantage of this method is its fast computation speed, but the disadvantage is the possibility of hash collisions. Second, TF-IDF-based deduplication technology converts text into a vector representation and then uses the TF-IDF algorithm to calculate the similarity between texts. The advantage of this method is that it can handle long texts, but the disadvantage is that it is not effective for short texts. Third, string matching-based deduplication technology utilizes string matching algorithms, such as exact string matching and fuzzy regular expression matching. The advantage of this method is high accuracy, but the disadvantage is high computational complexity and the inability to handle non-text objects such as images.

[0004] In bidding activities, due to the enormous economic interests involved, bid rigging and collusion frequently occur. These behaviors disrupt the normal market competition order, damage the economic interests of the tendering party or other bidders, and are often accompanied by commercial bribery, making them extremely harmful. Therefore, it is essential to detect bid rigging and collusion in tender documents. However, existing plagiarism detection algorithms rarely take into account the special characteristics of bidding activities and only check the text for plagiarism.

[0005] Patent document CN111144116A discloses a method and apparatus for extracting structured knowledge from documents. The method includes acquiring a document to be processed; selecting appropriate extraction rules based on the characteristics of the document; extracting knowledge points from the document according to the selected extraction rules; and performing pre-defined processing on the extracted knowledge points to complete the extraction of structured knowledge from the document. However, this invention does not fully consider the characteristics of bidding activities, and fails to remove reasonable references to bidding materials from the repetitive content of the bid. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for intelligent extraction of key knowledge points from docx format documents.

[0007] A method for intelligent extraction of key knowledge points from docx format documents according to the present invention includes: Step S1: Extract the bidding material information and remove references to the bidding materials; Step S2: Randomly select a pair of tender documents, A and B, from the combination. Using B and the tender materials as references, check for duplicates in document A. Step S3: Check the main text of the paragraph for plagiarism. Step S4: Check the attributes of tables, images, and documents for duplicates.

[0008] Preferably, in step S1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Deduplication checks are performed on multiple sets of bid documents. Multiple sets of bid documents can be processed in parallel using multi-process or multi-threaded methods, or each set can be processed sequentially using a for loop. The choice of method depends on hardware resources and business requirements. A set of bid documents is compiled from... Randomly select a pair of tender documents, file A and file B; In step S2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

[0009] Preferably, in step S3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Step A1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Step A2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Step A3: Extract a text slice from the paragraph text, with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, increment both start and end by 1 and start the next loop. Step A4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, take a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Step A5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Step A6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original paragraph length, sort the elements of the sets in ascending order to get a list, and use the first to second to last elements of the list as the starts variable, and the second to last elements as the ends variable. Step A7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attribute information, as well as the image information and corresponding position information that may be included, and the page break information that may be included, and clear the Paragraph object. Step A8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Step A9: If start is an element of the starts set, then the slice is a duplicate slice; in order to remove duplicates caused by the reference in the tender document, repeat the logic of steps A1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Step A10: For the sub-slice returned by the slice, if the sub-slice exists in the character variable of the tender document, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, construct a Run object, which is consistent with the logic of step A8, set the font color to the custom color of the duplicate part, and increment the number of duplicates in the paragraph body stage of the statistical result by 1. Step A11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

[0010] Preferably, in step S4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; The table is checked for duplicates. All Table nodes are retrieved from document.xml. The status variable is updated to "Table Stage", the total number of steps is the number of Table nodes, and the previous step count is 0. A for loop iterates through all Table nodes. For each Table node, each Cell node is traversed row by row. For a single Cell node, the text sequence contained in the Cell node is extracted. If the text sequence exists in file B and does not exist in the bidding document character variable, the font color of the Cell node is set to a custom duplicate text identifier color, and the number of repeated titles in the statistical results is incremented by 1. If the text sequence exists in file B and exists in the bidding document character variable, the number of title references in the statistical results is incremented by 1. After each Table node is calculated, the current step count is incremented by 1. The system performs image deduplication, inserting custom prompt paragraphs before the Paragraph nodes containing the image objects to be tagged; it retrieves all InlineShape nodes from document.xml, obtains the rId attribute value of each InlineShape node, and retrieves the corresponding image data from the media folder; it updates the state variables to the image stage, the total number of steps to the number of InlineShapes, and the current step count to 0; iterates through all InlineShapes using a for loop, and for a single InlineShape, it retrieves the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file image object set, but not equal to the blob value of any image in the bidding document image object set, it increments the image duplication count index by 1; if the blob value is equal to the blob value of any image in the B file image object set, and equal to the blob value of any image in the bidding document image object set, it increments the image reference count by 1; after processing each InlineShape node, it increments the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, unit, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False. Preferably, a multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Files A and B are returned with user-defined colors for user verification.

[0011] A key knowledge point extraction system based on docx format documents, provided by the present invention, includes: Module M1: Extracts information about bidding materials and removes references to those materials; Module M2: Randomly select a pair of tender documents, A and B, from the combination, and use B and the tender materials as references to check for duplicates in document A; Module M3: Performs plagiarism checks on the main text of paragraphs; Module M4: Performs duplicate checks on table, image, and document attributes.

[0012] Preferably, in module M1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Deduplication checks are performed on multiple sets of bid documents. Multiple sets of bid documents can be processed in parallel using multi-process or multi-threaded methods, or each set can be processed sequentially using a for loop. The choice of method depends on hardware resources and business requirements. A set of bid documents is compiled from... Randomly select a pair of tender documents, file A and file B; In module M2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

[0013] Preferably, in module M3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Module B1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Module B2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Module B3: Extracts a text slice from the paragraph text with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, both start and end are incremented by 1, and the next loop begins. Module B4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, retrieve a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Module B5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Module B6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original segment length, sort the elements in the sets in ascending order to get a list, and use the first to second to last elements in the list as the starts variable, and the second to last elements as the ends variable. Module B7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attributes, as well as image information and corresponding position information that may be included, and page break information that may be included, and clear the Paragraph object. Module B8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Module B9: If start is an element of the starts set, then the slice is a duplicate slice; in order to eliminate duplicates caused by the reference in the tender document, repeat the logic of modules B1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Module B10: For the sub-slice returned by the slice, if the sub-slice exists in the character variable of the tender document, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, a Run object is constructed, consistent with the logic of Module B8, the font color is set to the custom color of the duplicate part, and the number of duplicates in the paragraph body stage of the statistical result is incremented by 1. Module B11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

[0014] Preferably, in module M4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; The table is checked for duplicates. All Table nodes are retrieved from document.xml. The status variable is updated to "Table Stage", the total number of steps is the number of Table nodes, and the previous step count is 0. A for loop iterates through all Table nodes. For each Table node, each Cell node is traversed row by row. For a single Cell node, the text sequence contained in the Cell node is extracted. If the text sequence exists in file B and does not exist in the bidding document character variable, the font color of the Cell node is set to a custom duplicate text identifier color, and the number of repeated titles in the statistical results is incremented by 1. If the text sequence exists in file B and exists in the bidding document character variable, the number of title references in the statistical results is incremented by 1. After each Table node is calculated, the current step count is incremented by 1. The system performs image deduplication, inserting custom prompt paragraphs before the Paragraph nodes containing the image objects to be tagged; it retrieves all InlineShape nodes from document.xml, obtains the rId attribute value of each InlineShape node, and retrieves the corresponding image data from the media folder; it updates the state variables to the image stage, the total number of steps to the number of InlineShapes, and the current step count to 0; iterates through all InlineShapes using a for loop, and for a single InlineShape, it retrieves the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file image object set, but not equal to the blob value of any image in the bidding document image object set, it increments the image duplication count index by 1; if the blob value is equal to the blob value of any image in the B file image object set, and equal to the blob value of any image in the bidding document image object set, it increments the image reference count by 1; after processing each InlineShape node, it increments the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, unit, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False. Preferably, a multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Files A and B are returned with user-defined colors for user verification.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Based on this invention, users can perform weighted calculations on duplicate parts of the statistical results for specific business scenarios to obtain a total score. Based on the total score and established rules, automated evaluation and identification of bid rigging and collusion can be achieved. Alternatively, users can directly use the "custom color-coded" annotated bid documents returned by this invention to manually verify the highlighted sections in the bid documents, avoiding manual sentence-by-sentence comparison and greatly improving the efficiency of bid review.

[0016] 2. This invention fully considers the characteristics of bidding activities, eliminating reasonable references to bidding materials from duplicate content in bids, thus making the statistical results more accurate. Furthermore, the detection results also consider image objects and document attribute objects, making it more comprehensive than simple text-based plagiarism detection technology.

[0017] 3. This invention provides rich and user-friendly visualization features, including progress bars, statistical result display, and personalized highlighting of duplicate sections. Furthermore, it supports flexible multi-process / multi-threaded parallel computing methods, which can accelerate the speed of plagiarism detection and parsing. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A diagram illustrating a visualization of the progress information. Figure 2 An example diagram for visualizing statistical results; Figure 3 This is an example illustration of inserting a prompt paragraph before a repeating image paragraph. Detailed Implementation

[0019] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0020] Example 1: This invention proposes a novel detection algorithm, the innovation of which is reflected in the following aspects: (1) Fully consider the characteristics of bidding activities. In bidding activities, the tendering party usually issues a tender document, and the bidder submits a tender document based on the information and requirements in the tender document. In the process of writing the tender document, the bidder often quotes some content from the tender document, such as document outlines, table templates, etc. It is reasonable for the bidder to quote content from the tender document, so these repeated fragments should not be included in the statistical results.

[0021] (2) The processing of image objects has been increased. Existing detection technologies often only consider text information. However, if the same image, such as qualification certificate image, is detected in different tender documents, it can be largely determined that there is illegal bidding behavior.

[0022] (3) The detection of document attribute values ​​has been added. Existing detection technologies often only consider the main information of the document and ignore the document attribute information, such as the document template, title, author, organization, last modifier, etc. If the document attribute value information in the tender document can be automatically compared, the accuracy of the detection results can be further improved.

[0023] (4) Visualization function has been added. The algorithm parsing progress, statistical results of the number of times the data is counted and the number of times it is cited can be visualized. Since this detection method is based on the docx document format, duplicate information can be flexibly marked in the original tender document so that the tenderer can quickly locate and verify the duplicates.

[0024] According to the present invention, a method for intelligent extraction of key knowledge points from docx format documents is provided, such as... Figures 1-3 As shown, it includes: Step S1: Extract the bidding material information and remove references to the bidding materials; Specifically, in step S1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Deduplication checks are performed on multiple sets of bid documents. Multiple sets of bid documents can be processed in parallel using multi-process or multi-threaded methods, or each set can be processed sequentially using a for loop. The choice of method depends on hardware resources and business requirements. A set of bid documents is compiled from... Randomly select a pair of tender documents, file A and file B; Step S2: Randomly select a pair of tender documents, A and B, from the combination. Using B and the tender materials as references, check for duplicates in document A. In step S2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

[0025] Step S3: Check the main text of the paragraph for plagiarism. Specifically, in step S3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Step A1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Step A2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Step A3: Extract a text slice from the paragraph text, with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, increment both start and end by 1 and start the next loop. Step A4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, take a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Step A5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Step A6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original paragraph length, sort the elements of the sets in ascending order to get a list, and use the first to second to last elements of the list as the starts variable, and the second to last elements as the ends variable. Step A7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attribute information, as well as the image information and corresponding position information that may be included, and the page break information that may be included, and clear the Paragraph object. Step A8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Step A9: If start is an element of the starts set, then the slice is a duplicate slice; in order to remove duplicates caused by the reference in the tender document, repeat the logic of steps A1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Step A10: For the sub-slice returned by the slice, if the sub-slice exists in the character variable of the tender document, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, construct a Run object, which is consistent with the logic of step A8, set the font color to the custom color of the duplicate part, and increment the number of duplicates in the paragraph body stage of the statistical result by 1. Step A11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

[0026] Step S4: Check the attributes of tables, images, and documents for duplicates.

[0027] Specifically, in step S4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; The table is checked for duplicates. All Table nodes are retrieved from document.xml. The status variable is updated to "Table Stage", the total number of steps is the number of Table nodes, and the previous step count is 0. A for loop iterates through all Table nodes. For each Table node, each Cell node is traversed row by row. For a single Cell node, the text sequence contained in the Cell node is extracted. If the text sequence exists in file B and does not exist in the bidding document character variable, the font color of the Cell node is set to a custom duplicate text identifier color, and the number of repeated titles in the statistical results is incremented by 1. If the text sequence exists in file B and exists in the bidding document character variable, the number of title references in the statistical results is incremented by 1. After each Table node is calculated, the current step count is incremented by 1. The system performs image deduplication, inserting custom prompt paragraphs before the Paragraph nodes containing the image objects to be tagged; it retrieves all InlineShape nodes from document.xml, obtains the rId attribute value of each InlineShape node, and retrieves the corresponding image data from the media folder; it updates the state variables to the image stage, the total number of steps to the number of InlineShapes, and the current step count to 0; iterates through all InlineShapes using a for loop, and for a single InlineShape, it retrieves the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file image object set, but not equal to the blob value of any image in the bidding document image object set, it increments the image duplication count index by 1; if the blob value is equal to the blob value of any image in the B file image object set, and equal to the blob value of any image in the bidding document image object set, it increments the image reference count by 1; after processing each InlineShape node, it increments the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, unit, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False. Specifically, a multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Files A and B are returned with user-defined color annotations for user verification.

[0028] Example 2: Example 2 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0029] This invention also provides an intelligent extraction system for key knowledge points from docx format documents. The intelligent extraction system for key knowledge points from docx format documents can be implemented by executing the process steps of the intelligent extraction method for key knowledge points from docx format documents. That is, those skilled in the art can understand the intelligent extraction method for key knowledge points from docx format documents as a preferred embodiment of the intelligent extraction system for key knowledge points from docx format documents.

[0030] A key knowledge point extraction system based on docx format documents, provided by the present invention, includes: Module M1: Extracts information about bidding materials and removes references to those materials; Specifically, in module M1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Deduplication checks are performed on multiple sets of bid documents. Multiple sets of bid documents can be processed in parallel using multi-process or multi-threaded methods, or each set can be processed sequentially using a for loop. The choice of method depends on hardware resources and business requirements. A set of bid documents is compiled from... Randomly select a pair of tender documents, file A and file B; Module M2: Randomly select a pair of tender documents, A and B, from the combination, and use B and the tender materials as references to check for duplicates in document A; In module M2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

[0031] Module M3: Performs plagiarism checks on the main text of paragraphs; Specifically, in module M3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Module B1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Module B2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Module B3: Extracts a text slice from the paragraph text with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, both start and end are incremented by 1, and the next loop begins. Module B4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, retrieve a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Module B5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Module B6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original segment length, sort the elements in the sets in ascending order to get a list, and use the first to second to last elements in the list as the starts variable, and the second to last elements as the ends variable. Module B7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attributes, as well as image information and corresponding position information that may be included, and page break information that may be included, and clear the Paragraph object. Module B8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Module B9: If start is an element of the starts set, then the slice is a duplicate slice; in order to eliminate duplicates caused by the reference in the tender document, repeat the logic of modules B1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Module B10: For the sub-slice returned by the slice, if the sub-slice exists in the character variable of the tender document, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, a Run object is constructed, consistent with the logic of Module B8, the font color is set to the custom color of the duplicate part, and the number of duplicates in the paragraph body stage of the statistical result is incremented by 1. Module B11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

[0032] Module M4: Performs duplicate checks on table, image, and document attributes.

[0033] Specifically, in module M4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; The table is checked for duplicates. All Table nodes are retrieved from document.xml. The status variable is updated to "Table Stage", the total number of steps is the number of Table nodes, and the previous step count is 0. A for loop iterates through all Table nodes. For each Table node, each Cell node is traversed row by row. For a single Cell node, the text sequence contained in the Cell node is extracted. If the text sequence exists in file B and does not exist in the bidding document character variable, the font color of the Cell node is set to a custom duplicate text identifier color, and the number of repeated titles in the statistical results is incremented by 1. If the text sequence exists in file B and exists in the bidding document character variable, the number of title references in the statistical results is incremented by 1. After each Table node is calculated, the current step count is incremented by 1. The system performs image deduplication, inserting custom prompt paragraphs before the Paragraph nodes containing the image objects to be tagged; it retrieves all InlineShape nodes from document.xml, obtains the rId attribute value of each InlineShape node, and retrieves the corresponding image data from the media folder; it updates the state variables to the image stage, the total number of steps to the number of InlineShapes, and the current step count to 0; iterates through all InlineShapes using a for loop, and for a single InlineShape, it retrieves the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file image object set, but not equal to the blob value of any image in the bidding document image object set, it increments the image duplication count index by 1; if the blob value is equal to the blob value of any image in the B file image object set, and equal to the blob value of any image in the bidding document image object set, it increments the image reference count by 1; after processing each InlineShape node, it increments the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, unit, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False. Specifically, a multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Files A and B are returned with user-defined color annotations for user verification.

[0034] Example 3: Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0035] This invention proposes an intelligent algorithm for extracting key knowledge points from docx format documents.

[0036] Step 1: Extract bidding material information to eliminate reasonable references to the bidding materials. First, extract the text portion by paragraph, extracting the text strings from the bidding documents. The bidding documents can be single or multiple, and non-docx document formats are supported. Then, concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant. Similarly, extract all image objects and store them in the bidding document image collection constant.

[0037] Step 2: Assuming there are n bid documents, then it is necessary to... Deduplication checking of multiple sets of bid documents. This can be done using multi-process / multi-threaded parallel processing of multiple sets of bid documents, or by using a for loop to process each set sequentially. The specific method used depends on hardware resources and business requirements. Taking a particular set of bid documents as an example, from the combination... Randomly select a pair of tender documents, assuming they are named Document A and Document B.

[0038] Step 3: Using file B and the bidding materials as references, perform a duplicate check on file A. First, extract text, image, and document attribute information from file B, i.e., initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in each Paragraph node to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to each Picture node, retrieve the image data from the media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable.

[0039] Step 4: Begin deduplication checking of file A. To visualize the progress information, initialize a status variable containing the current parsing stage, total number of steps, and current step number. To display the statistical results, initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times the bidding materials are referenced.

[0040] Step 5: Perform plagiarism checks on the main body of the paragraph. First, count the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body stage, with the number of non-heading Paragraph nodes equal to the total number of steps, currently at 0. According to CNKI's plagiarism detection standards, if a segment of 13 or more consecutive characters from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: 1. If the paragraph text length is less than the threshold of 13, exit directly and start processing the next Paragraph node.

[0041] 2. If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, as well as the start and end variables and set their values ​​to 0 and 13 respectively. Start the while loop. The condition for the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop.

[0042] 3. Extract text slices from the paragraph text, with start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, increment both start and end by 1 and start the next loop.

[0043] 4. If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true. In the sub-loop, keep the `start` variable unchanged and increment the `end` variable. Retrieve a new slice. If the new slice is in the character variable of file B, increment the `end` variable again. After incrementing `end`, if `end` is greater than or equal to the paragraph text length, add the values ​​of `start` and `end` to the `starts` and `ends` sets respectively, and break the sub-loop. If the new slice is not in the character variable of file B, add the values ​​of `start` and `end` to the `starts` and `ends` sets respectively, assign the value of `end` to `start`, and add the threshold value to `end` and assign it to `end`. Exit the sub-loop.

[0044] 5. After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop.

[0045] 6. After the outer while loop finishes executing, if the `starts` collection is empty, it means there are no duplicate segments, so return directly. If the `starts` collection is not empty, merge the `starts` and `ends` collections, add element 0 and the original paragraph length, sort the elements in ascending order to obtain a list, and use the first to second-to-last elements of the list as the `starts` variable, and the second to last elements as the `ends` variable. 7. Reformat the Paragraph object. Extract the paragraph style values, default font, font size, font color, and other attribute information of the Paragraph object, as well as any image information and its corresponding position information, and any page break information it may contain, and then clear the Paragraph object.

[0046] 8. Retrieve a pair of start and end values ​​from the start and end list variables one by one. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom highlight color, and set the font size and font size to the default font and font size. Then add the Run object to the Paragraph object.

[0047] 9. If `start` is an element of the `starts` set, it indicates that the slice is a duplicate slice. To eliminate duplicates caused by references in the tender document, the logic in steps 1-6 is repeated, except that the scope of the slice check is not the character variables in file B, but the character variables in the tender document, and the paragraph object text becomes the text of the current slice.

[0048] 10. For sub-slices returned by the slice, if the sub-slice exists in the tender document character variable, then the sub-slice is a referenced part, and the number of referenced tender materials in the paragraph body stage of the statistical results is incremented by 1. Otherwise, it is a duplicate part. For the indeed duplicate part, a Run object also needs to be constructed. The steps are the same as in step 8, except that the font color is set to the custom highlight color of the duplicate part, and the number of duplicates in the paragraph body stage of the statistical results is incremented by 1.

[0049] 11. Finally, if the data includes non-text data such as images or page breaks, these should be inserted back into their original positions to avoid losing the original information. Finally, add paragraph styles to the paragraph objects.

[0050] Step 6: Perform duplicate checking on the title text. Since titles are generally brief and summary text, they are not suitable for the CNKI standard; this algorithm directly uses exact matching. First, retrieve all heading-type Paragraph nodes from document.xml. The style value of a heading-type Paragraph is either Heading or Title. Update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0. Begin iterating over the heading-type Paragraph nodes. If the text of a Paragraph node exists in the B file's character variable but not in the bidding document's character variable, mark the paragraph with a custom repeating color and increment the repeat count in the heading stage in the statistics by 1. If it exists in both the B file's character variable and the bidding document's character variable, increment the reference count in the heading stage by 1. Repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete.

[0051] Step 7: Perform duplicate checking on the tables. First, retrieve all Table nodes from document.xml, update the state variable to "Table Stage," the total number of steps to the number of Table nodes, and the previous step count variable to 0. Begin a for loop to iterate through all Table nodes. For each Table node, iterate through each Cell node row by row. For a single Cell node, extract the text sequence contained within the Cell node. If the text sequence exists in file B and not in the tender document's character variable, set the font color within the Cell node to a custom duplicate text identifier color and increment the title duplication count in the statistics by 1. If the text sequence exists in file B and exists in the tender document's character variable, increment the title reference count in the statistics by 1. After calculating each Table node, increment the current step count by 1.

[0052] Step 8: Perform image deduplication. Since image objects are not easily marked with font colors, and for easier verification, this algorithm inserts a custom prompt paragraph before the Paragraph node containing the image object to be marked. First, retrieve all InlineShape nodes from document.xml, obtain the rId attribute value of each InlineShape node, and retrieve the corresponding image data from the media folder. Update the state variable to the image stage, the total number of steps to the number of InlineShapes, and the current step count to 0. Iterate through all InlineShapes using a for loop. For a single InlineShape, obtain the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file image object set, but not equal to the blob value of any image in the bidding document image object set, increment the image duplication count index in the statistical results by 1; if the blob value is equal to the blob value of any image in the B file image object set, and equal to the blob value of any image in the bidding document image object set, increment the image reference count by 1. After processing each InlineShape node, increment the current step count by 1.

[0053] Step 9: Perform duplicate checking on the document attributes. Since document attribute parsing is relatively fast, this algorithm does not consider progress information at this stage. Only the five fields of template, title, organization, author, and last modified author are considered. The five attribute values ​​of file A and file B are compared separately. If any pair of attribute values ​​satisfies the condition that "neither is an empty string and they are equal," the document attribute in the statistical result is marked as True; otherwise, it is marked as False.

[0054] Steps 10 and 3 through 9 involve checking for duplicates in file A using file B and the bidding materials as references. This algorithm employs a multi-process approach, with file B checked for duplicates in an independent process, using file A and the bidding materials as references. The two sub-processes perform parallel computation, and their status information is visualized as progress bars. The final statistical results are then updated in the main process and returned to the user. Additionally, a "user-defined color-annotated version" of files A and B is returned for user verification.

[0055] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0056] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for intelligent extraction of key knowledge points from docx format documents, characterized in that, include: Step S1: Extract the bidding material information and remove references to the bidding materials; Step S2: Randomly select a pair of tender documents, A and B, from the combination. Using B and the tender materials as references, check for duplicates in document A. Step S3: Check the main text of the paragraph for plagiarism. Step S4: Check for duplicates in tables, images, and document attributes; In step S3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Step A1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Step A2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Step A3: Extract a text slice from the paragraph text, with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, increment both start and end by 1 and start the next loop. Step A4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, take a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Step A5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Step A6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original paragraph length, sort the elements of the sets in ascending order to get a list, and use the first to second to last elements of the list as the starts variable, and the second to last elements as the ends variable. Step A7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attribute information, as well as the image information and corresponding position information that may be included, and the page break information that may be included, and clear the Paragraph object. Step A8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Step A9: If start is an element of the starts set, then the slice is a duplicate slice; in order to remove duplicates caused by the reference in the tender document, repeat the logic of steps A1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Step A10: For the sub-slice returned by the slice, if the sub-slice exists in the character variable of the tender document, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, construct a Run object, which is consistent with the logic of step A8, set the font color to the custom color of the duplicate part, and increment the number of duplicates in the paragraph body stage of the statistical result by 1. Step A11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

2. The intelligent extraction method for key knowledge points based on docx format documents according to claim 1, characterized in that, In step S1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Check the duplicate entries in the tender documents; Multiple sets of bid documents can be computed in parallel using multi-process or multi-threaded methods, or each set of bid documents can be computed sequentially using a for loop. The choice of method depends on hardware resources and business requirements; a set of bid documents is composed of... Randomly select a pair of tender documents, file A and file B; In step S2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

3. The intelligent extraction method for key knowledge points based on docx format documents according to claim 1, characterized in that, In step S4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; Check for duplicates in the table, retrieve all Table nodes from document.xml, update the status variable to the table stage, the total number of steps to the number of Table nodes, and the previous step count variable to 0; start a for loop to traverse all Table nodes, and for each Table node, traverse each Cell node in a row-wise manner. For a single Cell node, extract the text sequence contained in the Cell node. If the text sequence exists in file B and not in the tender document character variable, set the font color of the Cell node to a custom repeating text identifier color and increment the title repetition count in the statistical results by 1. If the text sequence exists in file B and exists in the tender document character variable, increment the title reference count in the statistical results by 1. After each Table node is calculated, increment the current step number by 1. The image is checked for duplicates, and a custom prompt paragraph is inserted before the Paragraph node containing the image object to be tagged; all InlineShape nodes are retrieved from document.xml, the rId attribute value of each InlineShape node is obtained, and the corresponding image data is retrieved from the media folder; Update the state variables to the image stage and the total number of steps to the number of InlineShapes, with the current step count being 0. Iterate through all InlineShapes using a for loop. For a single InlineShape, obtain the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file's image object set, but not equal to the blob value of any image in the bidding document's image object set, increment the image duplication count in the statistical results by 1. If the blob value is equal to the blob value of any image in the B file's image object set, and equal to the blob value of any image in the bidding document's image object set, increment the image reference count by 1. After processing each InlineShape node, increment the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, organization, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False.

4. The intelligent extraction method for key knowledge points based on docx format documents according to claim 1, characterized in that: A multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Return A and B files with annotations based on user-defined colors for user verification.

5. A system for intelligent extraction of key knowledge points from docx format documents, characterized in that, include: Module M1: Extracts information about bidding materials and removes references to those materials; Module M2: Randomly select a pair of tender documents, A and B, from the combination, and use B and the tender materials as references to check for duplicates in document A; Module M3: Performs plagiarism checks on the main text of paragraphs; Module M4: Performs duplicate checks on table, image, and document attributes; In module M3: Perform plagiarism checks on the main body of the paragraph, counting the number of non-heading Paragraph nodes in document.xml. Non-heading refers to Paragraphs whose style is not Heading or Title. Update the status information to the main body of the paragraph stage, with the number of non-heading Paragraph nodes representing the total number of steps, currently at 0. If a segment of a consecutive preset threshold of characters or more from file A appears in file B, then that string segment from file A is considered a duplicate string. Use a depth-first strategy to traverse and calculate all non-heading Paragraph nodes. The plagiarism detection algorithm for a single paragraph of main text is calculated according to the following steps, incrementing the current step count by 1 for each Paragraph node calculated: Module B1: If the paragraph text length is less than the threshold, exit directly and start processing the next Paragraph node; Module B2: If the paragraph text length is greater than or equal to the threshold, initialize the start and end collection variables, and set the start and end variables to 0 and the preset threshold respectively. Start the while loop. The condition of the while loop is start plus the threshold. If it is less than the paragraph text length, continue the loop; otherwise, interrupt the loop. Module B3: Extracts a text slice from the paragraph text with the start and end variables as the starting and ending positions. If the slice does not exist in the character variables of file B, both start and end are incremented by 1, and the next loop begins. Module B4: If the slice exists in the character variable of file B, start a sub-while loop with the loop condition always true; in the sub-loop, keep the start variable unchanged and increment the end variable by 1, retrieve a new slice, if the new slice is in the character variable of file B, then continue to increment the end variable by 1; after the end variable is incremented, if end is greater than or equal to the paragraph text length, add the start and end values ​​to the starts and ends sets respectively, and break the sub-loop; if the new slice is not in the character variable of file B, add the start and end values ​​to the starts and ends sets respectively, assign the end value to start, add the threshold value to end and assign it to end; exit the sub-loop; Module B5: After exiting the sub-loop, check if the end variable is greater than or equal to the paragraph length. If it is, exit the outer while loop; otherwise, continue executing the outer while loop. Module B6: After the outer while loop finishes executing, if the starts set is empty, it means there are no duplicate segments, so return directly; if the starts set is not empty, merge the starts and ends sets, add element 0 and the original segment length, sort the elements in the sets in ascending order to get a list, and use the first to second to last elements in the list as the starts variable, and the second to last elements as the ends variable. Module B7: Reformat the Paragraph object; extract the paragraph style values, default font, font size, font color attributes, as well as image information and corresponding position information that may be included, and page break information that may be included, and clear the Paragraph object. Module B8: Take a pair of start and end values ​​from the start and end list variables in turn. If start is not an element in the start collection, create a Run object, encapsulate the sliced ​​text into the text property of the Run object, set the font color of the Run object to a custom color, set the font size and font size to the default font and font size, and add the Run object to the Paragraph object. Module B9: If start is an element of the starts set, then the slice is a duplicate slice; in order to eliminate duplicates caused by the reference in the tender document, repeat the logic of modules B1-A6, the scope of the slice check is the character variable of the tender document, and the paragraph object text becomes the current slice text; Module B10: For the sub-slice returned by the slice, if the sub-slice exists in the tender document character variable, then the sub-slice is the reference part, and the number of referenced tender materials in the paragraph body stage of the statistical result is incremented by 1; otherwise, it is the duplicate part. For the duplicate part, a Run object is constructed, consistent with the logic of Module B8, the font color is set to the custom color of the duplicate part, and the number of duplicates in the paragraph body stage of the statistical result is incremented by 1. Module B11: If non-text data is included, insert the non-text data into its original position to avoid losing the original information; finally, add paragraph styles to the paragraph object.

6. The intelligent extraction system for key knowledge points from docx format documents according to claim 5, characterized in that, In module M1: Extract bidding material information and remove references to bidding materials; extract text portions by paragraph, extracting text strings from the bidding documents (single or multiple bidding documents, supporting non-docx document formats); concatenate all paragraph strings with paragraph delimiters to form a bidding document string constant; extract all image objects and store the image objects into a bidding document image collection constant. n bid documents are processed Deduplication checks are performed on multiple sets of bid documents. Multiple sets of bid documents can be processed in parallel using multi-process or multi-threaded methods, or each set can be processed sequentially using a for loop. The choice of method depends on hardware resources and business requirements. A set of bid documents is compiled from... Randomly select a pair of tender documents, file A and file B; In module M2: Using document B and the bidding materials as references, perform a duplicate check on document A; Extract text, image, and document attribute information from file B, and initialize an image collection variable, a text string variable, and a document attribute dictionary variable. For the text string information in file B, traverse all Paragraph nodes in document.xml using a depth-first strategy, and add the text and paragraph separators contained in the Paragraph nodes to the text string variable in turn. For the image information in file B, traverse all Picture nodes in document.xml, find the rId corresponding to the Picture node, retrieve the image data from media based on the rId, and add it to the image collection variable. For the document attribute information, extract the node information from core.xml and app.xml and update the document attribute dictionary variable. Perform a duplicate check on file A, initialize a state variable containing the current parsing stage, total number of steps, and current step; initialize a statistical result variable containing the parsing stage, number of repetitions, and number of times bidding materials are referenced.

7. The intelligent extraction system for key knowledge points from docx format documents according to claim 5, characterized in that, In module M4: Perform duplicate checking on the title text; use exact matching to retrieve all heading-type Paragraph nodes from document.xml, with the style value of the heading-type Paragraph being either Heading or Title; update the state variable to the heading text stage, the total number of steps to the total number of heading-type Paragraph nodes, and the current step count to 0; iterate through the heading-type Paragraph nodes, if the text of the Paragraph node exists in the character variable of file B but not in the character variable of the tender document, mark the paragraph with a custom repeating color, and increment the repeat count of the heading stage in the statistics by 1; if it exists in the character variable of file B and also in the character variable of the tender document, increment the reference count of the heading stage by 1; repeat the loop process, incrementing the current step count by 1 with each iteration, until the loop is complete; Check for duplicates in the table, retrieve all Table nodes from document.xml, update the status variable to the table stage, the total number of steps to the number of Table nodes, and the previous step count variable to 0; start a for loop to traverse all Table nodes, and for each Table node, traverse each Cell node in a row-wise manner. For a single Cell node, extract the text sequence contained in the Cell node. If the text sequence exists in file B and not in the tender document character variable, set the font color of the Cell node to a custom repeating text identifier color and increment the title repetition count in the statistical results by 1. If the text sequence exists in file B and exists in the tender document character variable, increment the title reference count in the statistical results by 1. After each Table node is calculated, increment the current step number by 1. The image is checked for duplicates, and a custom prompt paragraph is inserted before the Paragraph node containing the image object to be tagged; all InlineShape nodes are retrieved from document.xml, the rId attribute value of each InlineShape node is obtained, and the corresponding image data is retrieved from the media folder; Update the state variables to the image stage and the total number of steps to the number of InlineShapes, with the current step count being 0. Iterate through all InlineShapes using a for loop. For a single InlineShape, obtain the blob value of the corresponding image object. If the blob value is equal to the blob value of any image in the B file's image object set, but not equal to the blob value of any image in the bidding document's image object set, increment the image duplication count in the statistical results by 1. If the blob value is equal to the blob value of any image in the B file's image object set, and equal to the blob value of any image in the bidding document's image object set, increment the image reference count by 1. After processing each InlineShape node, increment the current step count by 1. Perform duplicate checking on the document attributes, considering the template, title, organization, author, and last modified author. Compare the five attribute values ​​of file A and file B respectively. If any pair of attribute values ​​satisfies the condition that neither of them is an empty string and they are equal, mark the document attribute in the statistical result as True; otherwise, mark it as False.

8. The intelligent extraction system for key knowledge points from docx format documents according to claim 5, characterized in that: A multi-process approach is adopted. In an independent process, file B is checked for duplicates using file A and the bidding materials as references. The two sub-processes perform parallel calculations, and the status information of the two processes is visualized in the form of progress bars. The final statistical results are updated to the main process and returned to the user. Return A and B files with annotations based on user-defined colors for user verification.

Images