A method, system and program product for segmenting and aggregating query of engineering bidding documents

Abstract

The present invention belongs to the field of document analysis technology, and specifically discloses a method, system and program product for segmentation and aggregation query of engineering bidding documents, which identifies and analyzes the engineering bidding documents, splits them into several content blocks for word segmentation and vectorization, and then performs word segmentation and vectorization on the query content of the corresponding query subject, and then performs vector correlation calculation between the query and the blocks, selects the matching blocks for combination, and obtains the query results of the corresponding subject, so as to realize efficient and accurate content query of engineering bidding documents. Compared with the traditional manual reading and comprehension method, the present invention can restore the original content of the engineering bidding documents efficiently and at low cost, saves the time of manual reading, greatly improves work efficiency, and can prevent the situation of manual analysis errors, and to a certain extent avoid the written bidding documents from being cancelled, and meet the timeliness and accuracy requirements of users for the analysis of engineering bidding documents.

Description

Technical Field

[0001] The present invention belongs to the technical field of document analysis, and specifically relates to a method, system and program product for segmenting and aggregating query of engineering bidding documents. Background Art

[0002] The project bidding document is the outline of the bidding project construction, the basis for the construction unit to implement the project construction, and provides all the information required for the bidding unit to participate in the bidding. The quality and depth of the project bidding document are related to the success or failure of the entire bidding work, and the complexity of the project bidding document depends on the nature and scale of the bidding project.

[0003] For bidding projects with complex and large scale, the content of the bidding documents is complicated and often hundreds of pages long. If the bidding units use the traditional pure manual reading and comprehension method to parse them, they will incur huge manpower costs, the work efficiency will be very low, and the probability of human errors is high, which can easily lead to the situation where the bidding documents are cancelled, making it difficult to meet the requirements of timeliness and accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system and program product for segmenting and aggregating query of engineering bidding documents to solve the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, a method for segmenting and aggregating query of engineering bidding documents is provided, comprising:

[0007] Obtain the project bidding documents, perform text recognition on the project bidding documents, and obtain the corresponding content of the project bidding documents;

[0008] Perform text analysis on the content of the project bidding documents and divide the content of the project bidding documents into several content blocks;

[0009] Perform word segmentation processing on each content block, and perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block;

[0010] Retrieving each query content of the set query subject, performing word segmentation processing on each query content, and performing vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content;

[0011] Calculate the attention weight parameter of each content block relative to each query content according to the query vector corresponding to each query content and the block vector corresponding to each content block;

[0012] The content blocks corresponding to the largest number of attention weight parameters exceeding the set threshold are used as matching content blocks;

[0013] The matching contents are aggregated into blocks to obtain a block content combination of a set query subject, and the block content combination of the set query subject is stored and output.

[0014] In one possible design, the method further includes:

[0015] Obtaining a user's query statement, and performing semantic recognition and query subject matching based on the user's query statement to determine a number of corresponding query subjects;

[0016] The determined block content combinations of each query topic are recombined to obtain corresponding multi-topic block recombining results, and the multi-topic block recombining results are output.

[0017] In a possible design, the text of the project bidding document is parsed to divide the project bidding document into several content blocks, including:

[0018] Perform text parsing on the contents of the engineering bidding documents, and divide the contents of the engineering bidding documents into natural paragraphs, tables, pictures and / or page boundaries to obtain a number of content blocks;

[0019] The metadata corresponding to each content block is obtained, and each metadata is added to the corresponding content block.

[0020] In one possible design, for a content block containing a mathematical formula, the method further includes:

[0021] The mathematical formula in the content block is converted into formatted representation data through a set mathematical formula recognition tool.

[0022] In a possible design, each content block is word-segmented, and the word-segmented result is vectorized to obtain a block vector corresponding to each content block, including:

[0023] Performing word segmentation processing on each content block, dividing the content block into an independent element sequence, wherein the independent element sequence includes a plurality of independent elements;

[0024] The text vectorization model is used to vectorize the independent element sequence to obtain the block vector corresponding to each content block.

[0025] In a possible design, the step of calculating the attention weight parameter of each content block relative to each query content according to the query vector corresponding to each query content and the block vector corresponding to each content block includes:

[0026] Substitute the query vector corresponding to each query content and the block vector corresponding to each content block into the preset attention weight formula for traversal calculation to obtain the attention weight parameter of each content block relative to each query content. The attention weight formula is:

[0027]

[0028] Among them, i represents the label of the block vector corresponding to the content block, j represents the label of the query vector corresponding to the query content, and α ij Characterizes the attention weight parameter of the block vector corresponding to the i-th content block relative to the query vector corresponding to the j-th query content, k i Represents the block vector corresponding to the i-th content block, q j The query vector corresponding to the j-th query content, d k Represents the dimension of the corresponding block vector, and softmax() represents the normalized exponential function.

[0029] In a possible design, the block aggregation of each matching content to obtain a block content combination of a set query subject includes:

[0030] Combine the matching content blocks according to the set combination strategy to obtain an initial content combination;

[0031] The initial content combination is subjected to redundancy removal processing, semantic coherence processing and / or segmentation processing to obtain a block content combination of a set query subject.

[0032] In a second aspect, a system for segmenting and aggregating queries of engineering bidding documents is provided, including a text recognition unit, a text decomposition unit, a block processing unit, a query processing unit, a parameter calculation unit, a block matching unit and a block combination unit, wherein:

[0033] A text recognition unit is used to obtain the engineering bidding documents and perform text recognition on the engineering bidding documents to obtain the corresponding engineering bidding document contents;

[0034] A text decomposition unit is used to parse the contents of the engineering bidding document and divide the contents of the engineering bidding document into several content blocks;

[0035] A block processing unit is used to perform word segmentation processing on each content block, and to perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block;

[0036] A query processing unit, used to retrieve each query content of the set query subject, perform word segmentation processing on each query content, and perform vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content;

[0037] A parameter calculation unit, used to calculate an attention weight parameter of each content block relative to each query content according to a query vector corresponding to each query content and a block vector corresponding to each content block;

[0038] A block matching unit, used to take content blocks corresponding to a plurality of attention weight parameters exceeding a set threshold and having the largest value as matching content blocks;

[0039] The block combination unit is used to aggregate the matching content blocks to obtain the block content combination of the set query subject, and store and output the block content combination of the set query subject.

[0040] In the third aspect, a system for segmenting and aggregating engineering bidding documents is provided, including:

[0041] A memory for storing instructions;

[0042] A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0043] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored on the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any one of the methods described in the first aspect. In addition, a computer program product is provided, and when the computer program product is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0044] Beneficial effects: The present invention identifies and parses the engineering bidding documents, splits them into several content blocks for word segmentation and vectorization, then performs word segmentation and vectorization on the query content of the corresponding query subject, and then calculates the vector correlation between the query and the blocks, screens the matching blocks for combination, and obtains the query results of the corresponding subject, so as to achieve efficient and accurate content query of the engineering bidding documents. Compared with the traditional manual reading and comprehension method, the present invention can restore the original content of the engineering bidding documents efficiently and at low cost, saves manual reading time, greatly improves work efficiency, and can eliminate the situation of manual analysis errors, and to a certain extent avoid the written bidding documents from being rejected, meeting the user's requirements for timeliness and accuracy of engineering bidding document analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 This is a schematic diagram of the steps of the method in Example 1 of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the system in Example 2 of the present invention;

[0048] Figure 3 Schematic diagram of the system structure in Example 3 of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0050] It should be understood that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments can be understood according to specific circumstances.

[0051] In the following description, certain details are provided to facilitate a complete understanding of the example embodiments. However, it will be appreciated by those of ordinary skill in the art that the example embodiments may be implemented without these certain details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other embodiments, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0052] Embodiment 1:

[0053] This embodiment provides a method for segmenting and aggregating query of engineering bidding documents, such as Figure 1 As shown, the method comprises the following steps:

[0054] S1. Obtain the project bidding documents, perform text recognition on the project bidding documents, and obtain the corresponding project bidding document contents.

[0055] In the specific implementation, the engineering bidding document to be parsed is first obtained, and the engineering bidding document is a corresponding electronic file (such as doc, docx, PDF, HTML and other electronic files). Then, OCR (optical character recognition) technology can be used to perform text recognition on the engineering bidding document to extract the corresponding text information and obtain the corresponding engineering bidding document content. When the optical character recognition extracts the corresponding text information, the extracted text information can be checked for characters to ensure the accuracy of the optical character recognition results. The character check includes: 1. Character verification: Check whether the characters recognized by OCR conform to the expected character set (such as ASCII, Unicode). 2. Error correction: For characters that are recognized incorrectly, a dictionary or language model can be used to correct them. For example, if "the" is recognized as "teh", it can be corrected to "the" through a language model. 3. Determine character position information: retain the position information of the character in the page (such as coordinates, font size, color, etc.) for subsequent analysis.

[0056] S2. Perform text analysis on the contents of the project bidding documents and divide the contents of the project bidding documents into several content blocks.

[0057] In specific implementation, the content of the engineering bidding document is obtained through optical character recognition, and the text of the engineering bidding document content needs to be parsed, including layout recognition. Layout recognition involves identifying the structure and layout of the document, including paragraphs, titles, headers, footers, tables, pictures and other elements. The purpose of layout recognition is to segment the document content according to its logical structure for subsequent processing and analysis. The layout recognition process includes:

[0058] 1. Data preparation: Before performing layout recognition, you first need to prepare the original data of the document. This data usually includes the text recognized by OCR and its position information on the page (such as coordinates, font size, color, etc.). The data format includes text blocks, image blocks, and line information. Text block: Each text block contains one or more characters, as well as the position information of these characters on the page (such as the coordinates of the upper left corner and the lower right corner). Image block: If the document contains an image, the image block contains the position information of the image (such as a rectangular area). Line information: If the document contains a table or other line structure, the line information contains the position and direction of the line.

[0059] 2. Layout analysis: The purpose of layout analysis is to identify the overall structure of the document, including the partitioning of the page and different types of content areas. Page partitioning: Divide the page into different areas based on the position information of text blocks and image blocks. For example, the page can be divided into multiple rectangular areas by vertical and horizontal dividing lines. Content area differentiation: Classify each area based on the font size, position and content of the text block. For example, the area with a larger font may be a title, the area with a smaller font may be the body text, the area near the top of the page may be a header, and the area near the bottom of the page may be a footer.

[0060] 3. Paragraph detection: The purpose of paragraph detection is to identify the paragraph structure in a document, including line spacing analysis, paragraph boundary identification, and paragraph merging. Line spacing analysis: By analyzing the vertical distance between text blocks, the line spacing of the paragraph is identified. Usually, the line spacing between paragraphs is larger than the line spacing within a paragraph. Paragraph boundary identification: Based on the change in line spacing, the start and end positions of the paragraph are identified. For example, if the vertical distance between two text blocks is greater than a certain threshold, they are considered to belong to different paragraphs. Paragraph merging: For paragraphs that span across pages, they need to be merged into a complete paragraph.

[0061] 4. Title recognition: The purpose of title recognition is to recognize titles in a document, including chapter titles, subsection titles, etc. For example, font size analysis: By analyzing the font size of a text block, possible titles are identified. Usually, the font size of a title is larger than that of the text. Position analysis: By analyzing the position of a text block, possible titles are identified. For example, a text block near the top of the page may be a title. Content analysis: By analyzing the content of a text block, possible titles are identified. For example, titles usually contain keywords or phrases and are short.

[0062] 5. Header and footer recognition: The purpose of header and footer recognition is to identify the header and footer areas in the document. Including position analysis: By analyzing the position of the text block, possible headers and footers are identified. For example, the text blocks near the top and bottom of the page may be headers and footers. Content consistency analysis: By analyzing the content of the text block, possible headers and footers are identified. For example, headers and footers usually contain the same content and remain consistent between different pages.

[0063] 6. Table recognition: The purpose of table recognition is to recognize the table structure in the document, including the rows, columns, cells, etc. Including line detection: Identify the borders and internal lines of the table by analyzing the position and direction of the lines. Cell segmentation: According to the rows and columns of the table, the table is segmented into cells, each cell contains one or more text blocks. Table content extraction: Extract the text content in each cell and retain its position information in the table.

[0064] 7. Image recognition: The purpose of image recognition is to identify images in a document and separate them from text content. It includes image block detection: identifying images in a document by analyzing the position information of image blocks. Image content extraction: extracting the content of image blocks and retaining their position information in the page.

[0065] The document layout recognition process involves layout analysis, paragraph detection, title recognition, header and footer recognition, table recognition, and image recognition, which can ensure that the document structure and content information extracted in the end are accurate and complete. The results of layout recognition can be used for subsequent document content processing and analysis to segment the content of the project bidding document according to natural paragraphs, tables, pictures, and / or page boundaries to obtain several content blocks.

[0066] The initial segmented content blocks need to be added with necessary metadata, such as project name, project release date, etc., which are the natural attributes of the bidding documents collected during the file data recognition and collection process. Tables can be added with corresponding header texts, and pictures can be added with corresponding picture description texts, for further subject analysis and to avoid the concatenation of different file blocks in the future.

[0067] For mathematical formulas in content blocks, mathematical formulas in content blocks can be converted into formatted (such as LaTeX, MathML format) representation data through a set mathematical formula recognition tool (such as MathPix). The mathematical formula recognition process may include:

[0068] 1. Formula area detection and segmentation: First, we need to locate and segment the mathematical formula area. This can be achieved by using object detection technology (based on Faster R-CNN). This includes feature extraction: using a convolutional neural network to extract features from the image, and then locating the formula area through a classifier or regressor. Segmentation and segmentation: Segment the formula from the text through the extracted formula area.

[0069] 2. Character recognition: Each character or symbol in the formula needs to be recognized by OCR technology. In formula recognition, unlike ordinary text OCR, there are many kinds of mathematical symbols, and there are certain complex layout structures such as superscripts, subscripts, fractions, etc. Including symbol detection and recognition: Recurrent neural networks based on deep learning (such as LSTM, BiLSTM) are used to recognize mathematical symbols, including Latin letters, numbers, operation symbols (+, -, =, etc.) and special symbols (such as ∑, ∫, π, etc.). Utilization of context information: Since there may be contextual relationships between symbols in the formula, such as superscripts, subscripts, fractions, etc., this contextual information needs to be captured correctly. Sequence models (such as LSTM or Transformers) can better understand the logical structure of the formula by capturing contextual relationships.

[0070] 3. Formula structure analysis: Formula recognition is not just about identifying a single symbol, but more importantly, the hierarchical structure of the formula. For example, the hierarchical structure of complex formulas such as fractions, square roots, and matrices needs to be analyzed in order to be correctly converted into LaTeX, MathML, and other formats. Including tree structure analysis: Mathematical formulas can be regarded as a tree with a hierarchical structure. Each operator or symbol is a node of the tree. The structural relationship is established by analyzing the relative position (up, down, left, and right) between symbols, which can be completed by a parser (rule-based parsing or deep learning model). Analysis of formula structure: Recently, deep learning models based on graph structures (graph convolutional networks GCN) have been used to analyze formula structures, which can effectively handle the complex symbol arrangement and contextual relationship of formulas.

[0071] 4. Transcoding to LaTeX or MathML: Once the symbols and structure of the formula are recognized, they can be converted to the corresponding structured representation format LaTeX or MathML. Including symbol matching and format conversion: Each recognized symbol will be mapped to the corresponding LaTeX or MathML command, for example, "∑" will be converted to LaTeX's \sum command, and fractions will be converted to \frac{numerator}{denominator} format. Hierarchical symbol arrangement: The hierarchical relationship of complex formulas is expressed through nested LaTeX or MathML syntax.

[0072] 5. Tokenization: After the formula is successfully converted to LaTeX or MathML, the next step is to tokenize it. Tokenization refers to dividing the string into the smallest meaningful units to represent data. This process is to facilitate subsequent semantic analysis, automatic reasoning and other tasks. The tokenized representation data includes grammatical symbols: the LaTeX grammatical symbols in the formula are extracted as independent representation data, such as \sum, \frac, etc. Operands and operators: Operands (such as numbers, variables) and operators (such as addition, subtraction, multiplication and division) are also used as independent representation data.

[0073] S3. Perform word segmentation processing on each content block, and perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block.

[0074] In the specific implementation, each content block is word-segmented, and the content block is divided into an independent element sequence, where the independent element sequence includes a number of independent elements. The independent element sequence is then vectorized using a text vectorization model to obtain a block vector corresponding to each content block.

[0075] Tokenization is the process of dividing continuous content (such as text) into basic independent elements. For mathematical formulas, tokenization is equivalent to separating different symbols, operators, variables, etc. in the formula to form structured independent elements. A character sequence of a formula or a formula block, such as E = mc^2, tokenization is to decompose it into independent parts of E, =, m, c, ^, 2. In complex formulas, these basic units may include symbols such as +, -, *, / , =, ∑, ∫, ∞, etc.; variables and constants such as x, y, π, e, 1, 2, 3, etc.; function names such as sin, cos, log, exp, etc.; subscripts and matrix symbols such as ^, _, [], {}, \frac{}, etc. The result of tokenization is to convert the formula into a sequence of independent elements. For the tokenization of mathematical formulas, it is necessary not only to decompose the individual symbols, but also to consider the relationship between the symbols. The spatial relationship between symbols, such as subscripts and subscripts, fractions, and upper and lower bounds of integrals, needs to be represented in LaTeX syntax using specific symbols.

[0076] Vectorization is the process of converting raw data (such as text, images, and symbols) into a numerical form that can be represented by a vector. In mathematical formula processing, symbols, words, or entire formulas are converted into a high-dimensional vector (a point in a vector space). This numerical representation can capture information such as the semantics and contextual relationships of the symbol. After the word segmentation process is completed, each independent element (symbol, operator, variable, etc.) needs to be vectorized. Vectorization can use BERT or Transformers models for context-aware vectorization. Each symbol is not only encoded into a vector, but its vector representation also takes into account the contextual meaning of the symbol in the entire formula. For example, for the symbol x, it may have different meanings in different formulas, and the vector representation will be adjusted according to the context in which it is located. The result of vectorization is that each independent element after word segmentation is represented as a vector of a specific length, such as each symbol is vectorized into a 300-dimensional vector.

[0077] S4. Retrieve each query content of the set query subject, perform word segmentation processing on each query content, and perform vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content.

[0078] In specific implementation, the corresponding query topics can be pre-set, such as defining the core topics such as bidder qualification requirements, bidder performance requirements, bidder personnel requirements, and bid evaluation benchmark prices. Each query topic contains several different query contents, that is, corresponding to different query statements. These query contents can be obtained by manual setting or through a neural network model (such as by prompting the training of the large model DeepSeek to use the core vocabulary of the topic to generate different expression question samples of the same topic). Then, for the set query topic, each query content is segmented, and the segmentation results are vectorized to obtain the query vector corresponding to each query content. The process of segmentation and vectorization is the same as the processing method of content segmentation.

[0079] S5. Calculate the attention weight parameter of each content block relative to each query content according to the query vector corresponding to each query content and the block vector corresponding to each content block.

[0080] In the specific implementation, the query vector corresponding to each query content and the block vector corresponding to each content block are substituted into the preset attention weight formula for traversal calculation to obtain the attention weight parameter of each content block relative to each query content. The attention weight formula is:

[0081]

[0082] Among them, i represents the label of the block vector corresponding to the content block, j represents the label of the query vector corresponding to the query content, and α ij Characterizes the attention weight parameter of the block vector corresponding to the i-th content block relative to the query vector corresponding to the j-th query content, k i Represents the block vector corresponding to the i-th content block, q j The query vector corresponding to the j-th query content, d k Represents the dimension of the corresponding block vector, and softmax() represents the normalized exponential function.

[0083] S6. The content blocks corresponding to the largest number of attention weight parameters exceeding the set threshold are taken as matching content blocks.

[0084] In specific implementation, the corresponding attention weight parameters are arranged in descending order, and then the content blocks corresponding to the largest attention weight parameters that exceed the set threshold (when the attention weight parameter corresponding to a content block is higher than the preset threshold, it can be considered that the content block is highly relevant to the query topic) are taken as matching content blocks.

[0085] S7. Aggregate the matching content blocks to obtain a block content combination of the set query subject, and store and output the block content combination of the set query subject.

[0086] In specific implementation, the matching content blocks that are highly related to the same query subject can be aggregated together to obtain corresponding block content combinations for storage (building the corresponding knowledge base) and output. When aggregating the matching content blocks, the matching content blocks can be combined according to the set combination strategy (such as sorting) to obtain the initial content combination. Then, the initial content combination is post-processed accordingly to obtain the block content combination of the set query subject. Post-processing is a process of further optimizing the initial content combination to ensure that the result meets the application requirements, can more accurately reflect the subject content, and achieve the best coherence and relevance. Post-processing includes redundancy removal processing, semantic coherence processing and / or segmentation processing.

[0087] Redundancy removal processing. In the process of text block combination, redundant fragments are often generated, especially when there are repeated or similar content in the document. The steps to remove these redundant contents include the following methods: 1. Similarity calculation to remove redundancy, use cosine similarity to measure the similarity between multiple blocks, merge or remove highly similar information (such as similarity > 0.85), and only retain unique and representative content. 2. Keyword deduplication, based on the topic query vector or keyword list, calculate the frequency of important keywords in the block. If the frequency of a keyword is too high and it appears in multiple similar blocks, the redundancy can be deleted according to application requirements. 3. Sentence-level deduplication, the blocks can be further decomposed into sentences, and the sentences are deduplicated. The sentences are ranked by importance through TF-IDF technology, the most important sentences are retained, and duplicate information is removed.

[0088] Semantic coherence processing. After the blocks are combined, the contextual coherence of the fragments may be affected. Therefore, the coherence of the text can be enhanced in the following ways: 1. Language model fine-tuning: Use pre-trained language models (such as BERT, etc.) to correct the language naturalness of the blocks to ensure the coherence and fluency between the fragments. The model processing process may include: context-aware generation: Analyze the context of each fragment and generate transition sentences or sentences to make the transition between blocks smoother; sentence connection optimization: Automatically generate connecting words or phrases through natural language processing technology (Coherence Modeling) to enhance contextual logic. For example, transition words can be inserted between blocks. 2. Syntactic analysis: Use dependency parsing to check the dependency relationship between sentences, analyze the association between contexts, and make appropriate adjustments to the content according to the structure of the dependency tree. For example, if some fragments lack a subject or predicate, these elements can be added to make the sentences more coherent.

[0089] Segmentation processing. In some application scenarios, content segments need to be re-divided or merged to meet business needs. Mainly includes: 1. Topic clustering. If a block contains multiple topics, you can use the topic model (BERT Topic) to segment it, and then generate new sub-blocks. Use the clustering algorithm (K-Means) on the generated sub-blocks to merge blocks with similar topics. Each clustering result represents a complete topic-related content to avoid too many and scattered fragments. 2. Further segmentation based on length, segmentation strategy: If the block is too long, the long block can be further divided according to grammatical, semantic, or logical levels. For example, split according to the length of the sentence, or through the topic conversion signal (such as transition words) within the paragraph.

[0090] By storing the block content combination of the set query topic, the corresponding knowledge base can be constructed. When the user queries in the form of questions and answers, the user's query statement can be obtained, and semantic recognition and query topic matching can be performed based on the user's query statement, including question reconstruction (Question Rewriting), intent recognition (Intent Classification) and slot filling (Slot Filling) of the user's query statement, etc., to improve the user's question, make the question content specific, semantically derived and expression enriched, so as to determine the corresponding query topic based on natural language processing, and output the block content combination of the corresponding query topic. When multiple query topics are determined based on the user's query statement, the block content combination of each determined query topic can be recombined to obtain the corresponding multi-topic block recombined result, and the multi-topic block recombined result is output.

[0091] Compared with the traditional manual reading and comprehension method, the method of this embodiment can restore the original content of the engineering bidding document efficiently and at low cost, save the time of manual reading, greatly improve work efficiency, and can eliminate the situation of human parsing errors, and to a certain extent avoid the written bidding documents from being cancelled, thus meeting the user's requirements for the timeliness and accuracy of engineering bidding document parsing.

[0092] Embodiment 2:

[0093] This embodiment provides a system for segmenting and aggregating engineering bidding documents. Figure 2 As shown, it includes a text recognition unit, a text disassembly unit, a block processing unit, a query processing unit, a parameter calculation unit, a block matching unit and a block combination unit, wherein:

[0094] A text recognition unit is used to obtain the engineering bidding documents and perform text recognition on the engineering bidding documents to obtain the corresponding engineering bidding document contents;

[0095] A text decomposition unit is used to parse the contents of the engineering bidding document and divide the contents of the engineering bidding document into several content blocks;

[0096] A block processing unit is used to perform word segmentation processing on each content block, and to perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block;

[0097] A query processing unit, used to retrieve each query content of the set query subject, perform word segmentation processing on each query content, and perform vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content;

[0098] A parameter calculation unit, used to calculate an attention weight parameter of each content block relative to each query content according to a query vector corresponding to each query content and a block vector corresponding to each content block;

[0099] A block matching unit, used to take content blocks corresponding to a plurality of attention weight parameters exceeding a set threshold and having the largest value as matching content blocks;

[0100] The block combination unit is used to aggregate the matching content blocks to obtain the block content combination of the set query subject, and store and output the block content combination of the set query subject.

[0101] Embodiment 3:

[0102] This embodiment provides a system for segmenting and aggregating engineering bidding documents. Figure 3 As shown, at the hardware level, it includes:

[0103] Data interface, used to establish data connection between the processor and the external data terminal;

[0104] A memory for storing instructions;

[0105] The processor is used to read the instructions stored in the memory and execute the engineering bidding document segmentation and aggregation query method in Example 1 according to the instructions.

[0106] Optionally, the system further includes an internal bus, through which the processor, the memory and the data interface can be interconnected, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0107] The memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0108] Embodiment 4:

[0109] This embodiment provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method for segmenting and aggregating engineering bidding documents in Embodiment 1. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0110] This embodiment also provides a computer program product, which, when running on a computer, executes the engineering bidding document segmentation and aggregation query method in Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0111] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for segmenting and aggregating engineering bidding documents, characterized in that: include: Obtain the project bidding documents, perform text recognition on the project bidding documents, and obtain the corresponding content of the project bidding documents; Perform text analysis on the content of the project bidding documents and divide the content of the project bidding documents into several content blocks; Perform word segmentation processing on each content block, and perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block; Retrieving each query content of the set query subject, performing word segmentation processing on each query content, and performing vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content; Calculate the attention weight parameter of each content block relative to each query content according to the query vector corresponding to each query content and the block vector corresponding to each content block; The content blocks corresponding to the largest number of attention weight parameters exceeding the set threshold are used as matching content blocks; The matching contents are aggregated into blocks to obtain a block content combination of a set query subject, and the block content combination of the set query subject is stored and output.

2. A method for segmenting and aggregating engineering bidding documents according to claim 1, characterized in that: The method further comprises: Obtaining a user's query statement, and performing semantic recognition and query subject matching based on the user's query statement to determine a number of corresponding query subjects; The determined block content combinations of each query topic are recombined to obtain corresponding multi-topic block recombining results, and the multi-topic block recombining results are output.

3. The method for segmenting and aggregating engineering bidding documents according to claim 1 is characterized in that: The text parsing of the project bidding document content is performed to divide the project bidding document content into several content blocks, including: Perform text parsing on the contents of the engineering bidding documents, and divide the contents of the engineering bidding documents into natural paragraphs, tables, pictures and / or page boundaries to obtain a number of content blocks; The metadata corresponding to each content block is obtained, and each metadata is added to the corresponding content block.

4. A method for segmenting and aggregating engineering bidding documents according to claim 3, characterized in that: For a content block containing a mathematical formula, the method further includes: The mathematical formula in the content block is converted into formatted representation data through a set mathematical formula recognition tool.

5. The method for segmenting and aggregating engineering bidding documents according to claim 1 is characterized in that: The word segmentation process is performed on each content block, and the word segmentation process result is vectorized to obtain a block vector corresponding to each content block, including: Performing word segmentation processing on each content block, dividing the content block into an independent element sequence, wherein the independent element sequence includes a plurality of independent elements; The text vectorization model is used to vectorize the independent element sequence to obtain the block vector corresponding to each content block.

6. A method for segmenting and aggregating engineering bidding documents according to claim 1, characterized in that: The step of calculating the attention weight parameter of each content block relative to each query content according to the query vector corresponding to each query content and the block vector corresponding to each content block includes: Substitute the query vector corresponding to each query content and the block vector corresponding to each content block into the preset attention weight formula for traversal calculation to obtain the attention weight parameter of each content block relative to each query content. The attention weight formula is: Among them, i represents the label of the block vector corresponding to the content block, j represents the label of the query vector corresponding to the query content, and α ij Characterizes the attention weight parameter of the block vector corresponding to the i-th content block relative to the query vector corresponding to the j-th query content, k i Represents the block vector corresponding to the i-th content block, q j The query vector corresponding to the j-th query content, d k Represents the dimension of the corresponding block vector, and softmax() represents the normalized exponential function.

7. A method for segmenting and aggregating engineering bidding documents according to claim 1, characterized in that: The step of aggregating the matching contents into blocks to obtain a block content combination of the set query subject includes: Combine the matching content blocks according to the set combination strategy to obtain an initial content combination; The initial content combination is subjected to redundancy removal processing, semantic coherence processing and / or segmentation processing to obtain a block content combination of a set query subject.

8. A system for segmenting and aggregating engineering bidding documents, characterized in that: It includes a text recognition unit, a text disassembly unit, a block processing unit, a query processing unit, a parameter calculation unit, a block matching unit and a block combination unit, wherein: A text recognition unit is used to obtain the engineering bidding documents and perform text recognition on the engineering bidding documents to obtain the corresponding engineering bidding document contents; A text decomposition unit is used to parse the contents of the engineering bidding document and divide the contents of the engineering bidding document into several content blocks; A block processing unit is used to perform word segmentation processing on each content block, and to perform vectorization processing on the word segmentation processing result to obtain a block vector corresponding to each content block; A query processing unit, used to retrieve each query content of the set query subject, perform word segmentation processing on each query content, and perform vectorization processing on the word segmentation processing result to obtain a query vector corresponding to each query content; A parameter calculation unit, used to calculate an attention weight parameter of each content block relative to each query content according to a query vector corresponding to each query content and a block vector corresponding to each content block; A block matching unit, used to take content blocks corresponding to a plurality of attention weight parameters exceeding a set threshold and having the largest value as matching content blocks; The block combination unit is used to aggregate the matching content blocks to obtain the block content combination of the set query subject, and store and output the block content combination of the set query subject.

9. A system for segmenting and aggregating engineering bidding documents, characterized in that: include: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the engineering bidding document segmentation and aggregation query method described in any one of claims 1 to 7 according to the instructions.

10. A computer program product, characterized in that When the computer program product runs on a computer, the engineering bidding document segmentation and aggregation query method described in any one of claims 1 to 7 is executed.

Images