Bidding analysis method and system based on adaptive structured information extraction

Abstract

The application discloses a bidding analysis method and system based on adaptive structured information extraction and mainly relates to the technical field of natural language processing. The method comprises the following steps: collecting bidding technical specification files, performing data processing and direction labeling, and constructing a data set; constructing and training a rotation recognition model; based on the trained rotation recognition model and an open-source layout analysis model, automatically obtaining table contents from the bidding files; based on a large language model, sequentially performing text classification to filter standard technical parameter tables, structured information extraction to obtain parameter types and demand values, bidding product related information extraction, and bidding comparison analysis to generate a five-column comparison analysis table; and finally forming a bidding auxiliary analysis result statistical table containing parameter satisfaction degree statistics. The application has the beneficial effects that it realizes the improvement of bidding efficiency and accuracy, and avoids information omission.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically a bidding analysis method and system based on adaptive structured information extraction. Background Technology

[0002] In the bidding process, companies typically need to select products from their own product lines that meet the requirements based on the detailed parameters listed in the bidding technical specifications. Currently, this is generally done manually by reviewing product documents one by one and comparing technical parameters. This method not only consumes a lot of time and human resources, but also, due to the involvement of subjective judgment, is prone to errors and product selection mistakes. Its efficiency and accuracy are insufficient to meet the increasingly complex bidding demands.

[0003] In existing technologies, using large language models (LLMs) for automatic recognition and matching is considered a potential solution. However, tender technical specifications come in various formats (DOC, PDF, etc.), and some PDF documents contain scanned images or rotated tables, leading to frequent information omissions in traditional OCR technologies. Furthermore, the length of tender texts makes direct input into LLMs not only demanding in terms of video memory but also prone to missing key parameters or generating analysis errors, making it difficult to guarantee the reliability of the analysis results.

[0004] Therefore, there is an urgent need for a bidding auxiliary analysis method that can adapt to various document formats, avoid the defects of LLM long text processing, and achieve automated and high-precision parameter comparison, so as to solve the technical problems of low efficiency, incomplete information extraction and unstable analysis results in the existing technology. Summary of the Invention

[0005] The purpose of this invention is to provide a bidding analysis method and system based on adaptive structured information extraction, which improves bidding efficiency and accuracy while avoiding information omission.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In the bidding process, companies typically need to select products from their own product lines that meet the requirements based on the detailed parameters listed in the bidding technical specifications. Currently, this is generally done manually by reviewing product documents one by one and comparing technical parameters. This method not only consumes a lot of time and human resources, but also, due to the involvement of subjective judgment, is prone to errors and product selection mistakes. Its efficiency and accuracy are insufficient to meet the increasingly complex bidding demands.

[0007] In existing technologies, using large language models (LLMs) for automatic recognition and matching is considered a potential solution. However, tender technical specifications come in various formats (DOC, PDF, etc.), and some PDF documents contain scanned images or rotated tables, leading to frequent information omissions in traditional OCR technologies. Furthermore, the length of tender texts makes direct input into LLMs not only demanding in terms of video memory but also prone to missing key parameters or generating analysis errors, making it difficult to guarantee the reliability of the analysis results.

[0008] Therefore, there is an urgent need for a bidding auxiliary analysis method that can adapt to various document formats, avoid the defects of LLM long text processing, and achieve automated and high-precision parameter comparison, so as to solve the technical problems of low efficiency, incomplete information extraction and unstable analysis results in the existing technology.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention effectively solves the problem of incomplete information extraction caused by variable formats, table rotation, and scanned images in tender documents by constructing a dedicated rotation recognition model and layout analysis model, ensuring the integrity and accuracy of structured data acquisition. Based on this, the invention designs a step-by-step processing flow based on a large language model. Through text classification and filtering, structured information extraction, and targeted post-processing mechanisms, it successfully avoids the risk of parameter omissions and analysis errors caused by long text input. Ultimately, this invention achieves full automation from tender document parsing to product parameter comparison, freeing human effort from tedious repetitive work and significantly improving the speed of bidding response and the reliability of decision-making. Attached Figure Description

[0010] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram of the rotation recognition model architecture of the present invention; Figure 3 This is a flowchart of the table content extraction process of the present invention; Figure 4 This is a diagram of the bidding auxiliary analysis architecture of the present invention; Figure 5 This is a schematic diagram of the system structure of the present invention; Figure 6 This is a flowchart of the rotation recognition alternative method of the present invention. Detailed Implementation

[0011] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0012] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0013] Example 1: This embodiment uses the bidding process of a power company participating in the "Transmission Line Monitoring Device Procurement Project" as an example to explain the method flow of the present invention in detail. Figure 1 As shown, the method includes the following steps: Step S1: Construct the training dataset: Collect historical bidding technical specifications documents (including DOC and PDF formats), and convert each page into an image. Perform data augmentation on 30% of these images by randomly rotating them by +90 degrees, and label all images with the rotation direction (0 degrees, +90 degrees, -90 degrees) to construct a dataset. .

[0014] Step S2: Construct and train the rotation recognition model: like Figure 2 As shown, a rotation recognition model is constructed. The model employs the PP-LCNetV2 architecture, with its terminal classification layer consisting of three neurons, corresponding to 0 degrees, +90 degrees, and -90 degrees. The model is initialized by loading parameters pre-trained using SSLD and utilizing the dataset. and objective function ,(in As training samples, The first one-hot label One element, Iterative fine-tuning is performed to obtain the trained result. .

[0015] Step S3: Obtain the table content: like Figure 3 As shown, it specifically includes: First, determine the format of the tender document. If it's a DOCX file, extract the tables directly; if it's a DOC file, use LibreOffice to convert it to DOCX and then extract the data; if it's a PDF file, determine if it's a scanned document. The judgment logic is: if the text is empty, or if the following conditions are met... (in , For the first Page image width and height, Total number of pages If the function is an indicator function, then it is determined to be a scanned document.

[0016] Secondly, for scanned PDF documents, use The image orientation is corrected, and the open-source layout analysis model PP-PicoDet is used to detect and crop table regions.

[0017] Finally, PaddleOCR was used to perform OCR on the cropped table image, replacing line breaks with spaces to obtain a set of unstructured text. .

[0018] Step S4: Perform intelligent analysis based on LLM: First, build a prompt word template. Includes examples: technical parameter requirements, goods demand and supply quantity, others). Text is embedded one by one The data is then input into the Qwen2-72B model quantized with Int4 for multi-class classification. This text is retained only if the LLM outputs a "Technical Specifications Table". .

[0019] Secondly, construct prompt word templates. ,Will Embedded Next, input LLM and extract a Markdown table with three columns: serial number, parameter type, and tender parameter requirement value. Post-process the LLM output table: if the number of "|" characters in a row is 3, insert " / |" as a placeholder; if the number is less than 3 or greater than 4, delete the row.

[0020] Finally, construct the prompt word template. Embed product information and standardized parameter tables Then input LLM to generate a five-column comparison table containing the serial number, parameter type, bidding parameters, product parameters, and analysis results. The analysis results include "Satisfied", "Partially Satisfied", "Not Satisfied", and "Cannot be determined".

[0021] Step S5: Generate a statistical table: The fifth column of the comparison table is statistically analyzed, including the parameter types and total number corresponding to different evaluation results, to form a statistical table of bidding auxiliary analysis results, such as... Figure 4 As shown, the overall analysis architecture is illustrated.

[0022] like Figure 5 As shown in the figure, this embodiment also provides a bidding analysis system based on adaptive structured information extraction, including: The data construction module is used to collect bidding technical specifications documents, perform data processing and orientation labeling, and build training datasets. The model training module is used to build and train a rotation recognition model to classify the orientation of image content; The content acquisition module is used to automatically acquire table content from the bidding technical specification document based on the trained rotation recognition model and the open-source layout analysis model. The intelligent analysis module is used to perform text classification, structured information extraction, bidding product association information extraction, and bidding comparison analysis on the acquired table content based on a large language model. The results generation module is used to generate a statistical table of bidding auxiliary analysis results, which includes statistics on the satisfaction of product parameters, based on the results of the bidding comparison analysis.

[0023] The content acquisition module includes: The format determination unit is used to determine the format of the bidding technical specification document and perform corresponding parsing and conversion operations for different format documents; The image correction unit is used to convert the file into an image when the file is a scanned PDF, and to perform orientation detection and correction on the image using the rotation recognition model. The table cropping unit is used to perform layout analysis on the corrected image using the open-source layout analysis model, and to locate and crop out the table area. The character recognition unit is used to perform optical character recognition on the cropped table area to generate unstructured text.

[0024] The intelligent analysis module includes: The text classification unit is used to construct text classification prompts to drive the large language model to perform multi-classification of unstructured text in order to filter out the standard technical parameter table. The information extraction and post-processing unit is used to construct information extraction prompts, drive the large language model to extract parameter types and required values ​​from the filtered text, and perform structured repair and verification on the Markdown table output by the large language model. The comparative analysis unit is used to construct comparative analysis prompts. It inputs product information and extracted bidding parameters into a large language model and generates a comparative analysis table containing satisfaction assessment through comparative analysis.

[0025] Example 2: As an alternative to rotation recognition in Example 1, such as Figure 6 As shown, a multi-stage OCR discrimination method can be used: Perform OCR on the original image to obtain text. Count the number of Chinese characters ; like Then rotate the image 90 degrees clockwise and perform OCR to obtain... (statistics Rotate counterclockwise 90 degrees and perform OCR to obtain (statistics ); like The maximum value indicates a non-rotational condition. Maximum, determined to be -90 degrees; Maximum, determined as +90 degrees.

[0026] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A bidding analysis method based on adaptive structured information extraction, characterized in that, Includes the following steps: Step S1: Collect the bidding technical specification documents, process and label the data, and construct a training dataset; Step S2: Build and train a rotation recognition model for classifying the orientation of image content; Step S3: Based on the trained rotation recognition model and open-source layout analysis model, automatically obtain the table content from the bidding technical specification document; Step S4: Based on a large language model, perform text classification, structured information extraction, bidding product association information extraction, and bidding comparison analysis on the obtained table content in sequence; Step S5: Based on the results of the bidding comparison analysis, generate a statistical table of bidding auxiliary analysis results that includes statistics on the satisfaction of product parameters.

2. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, Step S1 specifically includes: converting each page of the collected historical bidding technical specification documents into an image, and selecting a certain proportion of the images for random rotation of +90 degrees to perform data augmentation; labeling the rotation direction of all image data, wherein the rotation direction includes 0 degrees, +90 degrees and -90 degrees, where +90 degrees represents a 90-degree counterclockwise rotation of the forward image, and -90 degrees represents a 90-degree clockwise rotation of the forward image, thereby constructing the training dataset.

3. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, In step S2, the rotation recognition model adopts the lightweight convolutional neural network architecture PP-LCNetV2, which consists of a standard convolutional module, several depthwise separable convolutional modules, several depthwise separable convolution + reparameterization + compression and activation modules, several depthwise separable convolution + reparameterization + residual connection modules, global mean pooling, terminal convolutional modules, and a fully connected classification layer. The standard convolution module consists of standard convolution, batch normalization, and ReLU activation function. The depthwise separable convolutional module consists of a depthwise convolutional module and a pointwise convolutional module; the depthwise convolutional module consists of depthwise convolution, batch normalization, and ReLU activation, and the pointwise convolutional module has a kernel size of [missing information]. The standard convolutional module; The depthwise separable convolution + reparameterization + compression and activation module is composed of a reparameterization module, a compression and activation module, and a separable pointwise convolution module in sequence; the reparameterization module is used to process the convolution kernels with sizes of... , and A depthwise convolutional module and a convolutional kernel with a size of The different receptive field features extracted by the depthwise convolution are fused together, and the input is... The output is , represented as: ； in, For depthwise convolution operations, Indicates the kernel size as The cascaded operation of the depthwise convolutional modules; when the stride of the reparameterization module is 1, it includes the convolutional kernel size. The deep convolution module; the compression and activation module is composed of global mean pooling, pointwise convolution, ReLU activation, pointwise convolution, Sigmoid activation, and channel attention fusion in sequence, wherein the channel attention fusion is expressed as: ； in, The output after channel attention fusion. As input to the compression and excitation module, The output is activated by Sigmoid; the separate pointwise convolution module consists of two pointwise convolutions with different numbers of convolution kernels, used to compress the channels to half of the original number of channels before expanding and restoring them; The depthwise separable convolution + reparameterization + residual connection module consists of a reparameterization module, a pointwise convolution module, and residual connections, and its final output is... Input for reparameterization module Output of the pointwise convolution module The sum, that is: ; The terminal convolution module consists of pointwise convolution, ReLU activation, and a Dropout layer in sequence, with the Dropout layer having a dropout probability of 0.

2.

4. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, Step S3 specifically includes: If the tender technical specification document is in DOCX format, then directly extract the table content and convert it to Markdown format; If the tender technical specification document is in DOC format, it should first be converted to DOCX format, and then the table content should be extracted and converted to Markdown format. If the tender technical specification document is in PDF format, the first step is to determine whether the document is a scanned copy. The determination logic is as follows: use a PDF processing library to retrieve the text and images from the document; if the text is empty, it is determined to be a scanned copy; if the image is wide... Larger than page width of And image height Greater than page height of The number of images is greater than the total number of pages in the file. Half of it, that is, satisfying: ； in, This is an indicator function; its value is 1 when the input is true and 0 otherwise. If so, it is determined to be a scanned document; If the document is not scanned, it is converted to DOCX format and the table content is extracted. If it is scanned, each page is converted into an image set, and each image in the image set is classified according to its orientation based on the rotation recognition model. If the classification result is +90 degrees or -90 degrees, the image is corrected. The corrected image is then analyzed based on the open-source layout analysis model, and the image is cropped according to the table category detection result, retaining the table area. Finally, optical character recognition is performed on the cropped table image to obtain an unstructured text set.

5. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, Step S4, the steps for text classification based on a large language model, include: Build a text category prompt template The unstructured text is embedded one by one into The text is then input into a large language model, which determines whether the text belongs to a standard technical parameter table. The text classification adopts a multi-classification strategy, with categories including standard technical parameter tables, goods demand and supply quantity tables, and three other categories. The text is determined to belong to the standard technical parameter table and is retained only if the large language model outputs a sequence of "technical parameter requirements table"; otherwise, it is discarded.

6. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, Step S4, which involves extracting and post-processing structured information based on a large language model, includes: Build information extraction prompt word templates Embed the preserved text The data is then input into a large language model, which automatically extracts the structured information of parameter types and their required values, and outputs it as a Markdown-formatted table. The table includes three columns: serial number, parameter type, and tender parameter required value. The post-processing of the Markdown formatted table is as follows: the table is divided into several rows by line breaks. Starting from the third row, if the number of delimiters "|" in the current row is 4, then the current row is determined to be a normal row containing 3 cells; if the number of delimiters "|" in the current row is 3, then the string " / |" is added after the second delimiter in the row as a cell placeholder; if the number of delimiters "|" in the current row is less than 3 or greater than 4, then the current row is deleted and the sequence number of the subsequent rows is updated.

7. The bidding analysis method based on adaptive structured information extraction according to claim 1, characterized in that, Step S4, the steps for extracting relevant information of the bidding products and conducting comparative analysis of bidding and tendering, include: embedding the Markdown table information of the product models to be compared and the post-processed standard technical parameter table into the comparative analysis prompt template. The data is then input into a large language model; the output of the large language model is a Markdown comparison and analysis table containing five columns, namely, serial number, parameter type, tender parameter requirement value, product parameter value, and whether the tender requirements are met; the decision result of "whether the tender requirements are met" includes four types: met, partially met, not met, and cannot be determined.

8. A bidding analysis system based on adaptive structured information extraction, used to implement the bidding analysis method based on adaptive structured information extraction as described in any one of claims 1-7, characterized in that, include: The data construction module is used to collect bidding technical specifications documents, perform data processing and orientation labeling, and build training datasets. The model training module is used to build and train a rotation recognition model to classify the orientation of image content; The content acquisition module is used to automatically acquire table content from the bidding technical specification document based on the trained rotation recognition model and the open-source layout analysis model. The intelligent analysis module is used to perform text classification, structured information extraction, bidding product association information extraction, and bidding comparison analysis on the acquired table content based on a large language model. The results generation module is used to generate a statistical table of bidding auxiliary analysis results, which includes statistics on the satisfaction of product parameters, based on the results of the bidding comparison analysis.

9. A bidding analysis system based on adaptive structured information extraction according to claim 8, characterized in that, The content acquisition module includes: The format determination unit is used to determine the format of the bidding technical specification document and perform corresponding parsing and conversion operations for different format documents; The image correction unit is used to convert the file into an image when the file is a scanned PDF, and to perform orientation detection and correction on the image using the rotation recognition model. The table cropping unit is used to perform layout analysis on the corrected image using the open-source layout analysis model, and to locate and crop out the table area. The character recognition unit is used to perform optical character recognition on the cropped table area to generate unstructured text.

10. A bidding analysis system based on adaptive structured information extraction according to claim 8, characterized in that, The intelligent analysis module includes: The text classification unit is used to construct text classification prompts to drive the large language model to perform multi-classification of unstructured text in order to filter out the standard technical parameter table. The information extraction and post-processing unit is used to construct information extraction prompts, drive the large language model to extract parameter types and required values ​​from the filtered text, and perform structured repair and verification on the Markdown table output by the large language model. The comparative analysis unit is used to construct comparative analysis prompts. It inputs product information and extracted bidding parameters into a large language model and generates a comparative analysis table containing satisfaction assessment through comparative analysis.

Images