A multi-modal document analysis system for industrial material intelligent identification

Abstract

The application is particularly a multi-modal document analysis system for industrial material intelligent identification, and relates to the technical field of industrial material information processing, comprising: a knowledge graph verification module, which is used for inputting a standardized mapping result into an industrial material knowledge graph to perform bidirectional verification and obtain a verification result. In the application, the standardized mapping mode of multi-strategy voting comprehensively utilizes the complementarity of three matching strategies of string similarity matching, semantic vector similarity retrieval and rule engine analysis, overcomes the problem of insufficient adaptability of a single matching strategy to non-standardized expressions such as synonymous writing, abbreviation and abbreviation, and improves the reliability of material name standardization.

Description

Technical Field

[0001] This invention relates to the field of industrial material information processing technology, and in particular to a multimodal document parsing system for intelligent identification of industrial materials. Background Technology

[0002] In industrial production and supply chain management, material documents are core vouchers for procurement, warehousing, outbound delivery, and reconciliation. Existing document recognition technologies typically treat documents as a unified text stream for OCR recognition and information extraction, performing overall processing on the text content in the document image before extracting fields.

[0003] However, industrial material documents typically include functional areas such as a header area, material details area, summary area, and remarks and signature area, with different areas carrying material information of different natures. Existing technologies do not fully utilize the unique layout structure and semantics of industrial material documents, leading to cross-area interference and field misalignment issues during field alignment. Furthermore, descriptions of material names, specifications, and models contain numerous synonyms, abbreviations, and other non-standardized expressions. Existing methods lack effective verification mechanisms after extracting material information, resulting in a high parsing error rate. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal document parsing system for intelligent identification of industrial materials in order to solve the above-mentioned problems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal document parsing system for intelligent identification of industrial materials includes: The layout partitioning recognition module is used to perform semantic partitioning recognition on industrial material document images to obtain the functional areas and their semantic category labels; The targeted information extraction module is used to extract targeted information for each functional area based on the semantic category labels of each functional area, and obtain a structured bill of materials. The standardization mapping module is used to perform standardization mapping on the original name descriptions of each material record in the bill of materials. It uses a multi-strategy voting method to determine the mapping result and obtains the standardized material name and mapping confidence. The knowledge graph verification module is used to input the standardized mapping results into the industrial material knowledge graph to perform bidirectional verification and obtain the verification results; The confidence assessment module is used to calculate multi-dimensional confidence scores for each material record and generate an analytical result report based on the multi-dimensional confidence scores.

[0006] Preferably, the semantic partitioning recognition of the layout is achieved through a layout detection model. The input layer of the layout detection model receives the image feature sequence of the document image and the corresponding text position encoding sequence. The output layer contains two parallel fully connected layers. One fully connected layer outputs the bounding box coordinates of each region, and the other fully connected layer outputs the semantic category label of each region. The partitioning confidence score is calculated for each functional region output by the layout detection model, and regions with a partitioning confidence score lower than the partitioning threshold are marked as partitioning uncertain.

[0007] Preferably, the step of performing targeted information extraction on each functional area includes: For the material details area, the table structure is identified to obtain the row and column structure of the table. Based on the row and column structure, row alignment processing is performed on each row of material records. Each row of material is decomposed into the fields of material name, specification model, unit of measurement, quantity, unit price, and amount. The row alignment processing uses prior knowledge of the layout as a constraint condition. For the document header area, a named entity recognition model is used to extract the supplier name, document number, and date information; For the summary area, template matching is performed based on predefined field templates to extract the total amount and tax amount numerical fields; For the remarks and signature areas, extract the text content as an additional explanation field and store it in the remarks field of the bill of materials.

[0008] Preferably, during the row alignment process, the text content of each cell is classified into numeric fields, text containing only numbers or decimals is classified into name fields, and text containing unit symbols is classified into unit fields. The field type classification result is cross-validated with the expected field type corresponding to the column index. If the two are inconsistent, the column classification of the cell is corrected.

[0009] Preferably, the standardization mapping includes: performing abbreviation expansion processing on the original name description, expanding the abbreviation or acronym into a complete standard name based on a thesaurus; performing synonym unification processing on the original name description, mapping names with the same meaning but different expressions to the same standard name; performing specification parsing processing on the specification parameters in the original description, parsing the specification expression into structured specification parameters; and performing model standardization processing on the model information in the original description, unifying model differences from different sources into the enterprise standard model system.

[0010] Preferably, in the multi-strategy voting method, when a candidate standard name is recommended by at least two strategies and has the highest weighted score, the candidate standard name is determined as the standardized mapping result; the mapping confidence is the value after normalization of the weighted score, and the value range is 0 to 1; the text encoder in the semantic vector similarity retrieval is a text encoder trained on industrial material corpus for domain adaptation, and the contrastive learning loss is used as the loss function to update the parameters.

[0011] Preferably, the forward verification includes: checking whether the standard material name in the standardized mapping result exists in the industrial material knowledge graph; if it does not exist, the material record is marked as a new material and the new material registration process is triggered; the reverse verification includes: checking whether the specification parameters in the standardized mapping result fall within the range of standard parameters corresponding to the standard material name in the industrial material knowledge graph; when the specification parameters exceed the range of standard parameters, the material record is marked as a suspected identification error and the standardized mapping is re-executed.

[0012] Preferably, a maximum number of remapping attempts is set for each material record. When the cumulative number of times a material record has been re-executed for standardized mapping reaches the maximum number of remapping attempts, the material record is directly marked as pending manual review.

[0013] Preferably, the multi-dimensional confidence score includes OCR recognition confidence. Page layout stability Field alignment integrity Standardized mapping voting score Knowledge graph validation pass rate Overall confidence score ,in Preset weights for each dimension, It is a dimension index and satisfies ; The overall confidence score Below the preset threshold Material records are marked as pending review. Correct results after review and confirmation are used as positive samples and fed back to the targeted information extraction process and standardized mapping process for incremental learning.

[0014] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention utilizes a standardized mapping method based on multi-strategy voting, combining the complementarity of three matching strategies: string similarity matching, semantic vector similarity retrieval, and rule engine parsing. This overcomes the problem of insufficient adaptability of a single matching strategy to non-standardized expressions such as synonyms, abbreviations, etc., and improves the reliability of material name standardization.

[0015] 2. This invention verifies the standardized mapping results from two aspects: the existence of materials and the rationality of specifications, through forward and reverse verification of industrial material knowledge graphs. This effectively filters out material information distortion caused by OCR misidentification or format parsing errors. Attached Figure Description

[0016] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0017] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0018] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0019] Example 1 Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0020] In this embodiment, it includes: In industrial production and supply chain management, material documents are core vouchers for procurement, warehousing, outbound shipments, and reconciliation. Existing document recognition technologies typically treat documents as a uniform text stream for OCR recognition and information extraction, failing to fully utilize the unique layout and semantics of industrial material documents. Industrial material documents usually include functional areas such as a header area, material details area, summary area, and remarks and signature area, with different areas carrying different types of material information. Furthermore, descriptions of material names, specifications, and models contain numerous synonyms, abbreviations, and other non-standardized expressions. Existing methods lack effective verification mechanisms after extracting material information, resulting in a high parsing error rate.

[0021] This embodiment proposes an industrial material document parsing method, which achieves accurate parsing of industrial material documents and reliable extraction of material information through semantic partitioning recognition of layout, targeted information extraction within partitions, standardized mapping of material names, bidirectional verification of knowledge graph, and confidence evaluation of parsing results.

[0022] An industrial material document parsing method according to an embodiment of the present invention includes the following modules: Module 1 performs semantic partitioning recognition on industrial material document images to obtain the functional areas and their semantic category labels.

[0023] The process involves acquiring the image of the industrial material document to be parsed, inputting it into the layout detection model, and outputting the geometric coordinates of each functional area in the document image along with its corresponding semantic category labels. The semantic category labels include the document header area, material details area, summary area, and remarks and signature area. Specifically, the document header area corresponds to the document type, document number, date, and supplier or customer information; the material details area corresponds to the material line list; the summary area corresponds to the total amount, amount in words, tax amount, and other summary data; and the remarks and signature area corresponds to additional instructions, approval signatures, and stamps.

[0024] It should be noted that the layout detection model is a fine-tuned version of LayoutLMv3. The input layer of LayoutLMv3 receives the image feature sequence of the document image and the corresponding text position encoding sequence. The output layer contains two parallel fully connected layers: one fully connected layer outputs the bounding box coordinates of each region (corresponding to the region bounding box regression task), and the other fully connected layer outputs the semantic category label of each region (corresponding to the semantic category classification task). During fine-tuning, industrial material document samples are used as training data, and the sum of the region bounding box regression loss and the semantic category classification cross-entropy loss is used as the loss function. The Adam optimization algorithm is employed to update the model parameters, enabling the layout detection model to learn the layout structure features of industrial material documents.

[0025] In this embodiment, to improve the adaptability of partition recognition to different document formats, the layout detection model introduces multi-format document samples during the training phase, covering layout differences between different suppliers and document types. During the inference phase, for each functional area output by the layout detection model, a partition confidence score is calculated for each area. Areas with partition confidence scores below the partition threshold are marked as "partition uncertain" for reference by subsequent modules.

[0026] Module 2 performs targeted information extraction on each functional area based on the semantic category labels of each functional area to obtain a structured bill of materials.

[0027] Different information extraction strategies are adopted for functional areas with different semantic categories.

[0028] For the material details area, table structure recognition is used to perform row and column detection on the tables within the material details area to obtain the table's row and column structure. Based on the row and column structure, row alignment processing is performed on each material record, breaking down each material record into structured fields such as material name, specifications, unit of measurement, quantity, unit price, and amount. During the row alignment process, prior layout knowledge is used as constraints. The prior layout knowledge includes: the material name is located in the leftmost column of the details area, and the specifications are located in the adjacent right column of the material name.

[0029] For the document header area, a named entity recognition (NAME) model is used to identify entities within the text, extracting key information such as supplier name, document number, and date. The NAME model is a sequence labeling model; its input layer receives the text sequence from the document header area, and its output layer is a fully connected layer that outputs the probability distribution of entity category labels for each position. The labeling system uses the BIO format. During training, the labeled document header text sequence is used as training data, and the sequence labeling cross-entropy loss is used as the loss function. The Adam optimization algorithm is used to update the parameters of the NAME model.

[0030] For the summary area, template matching is performed on the summary area text based on predefined field templates to extract numerical fields such as total amount and tax amount.

[0031] For the notes and signature areas, the text content within these areas is extracted as additional explanation fields. No structured field decomposition is performed; the text is directly stored in the notes field of the bill of materials in its original form.

[0032] After the extraction results from each functional area are aggregated, a structured bill of materials is output. Each material record in the bill of materials contains a raw description field directly extracted from the document.

[0033] It should be noted that table structure recognition refers to performing table detection and cell positioning on the image area of ​​the material details section, identifying the row and column boundaries of the table, and then determining the position of each cell and its corresponding row and column index. Row alignment processing refers to matching the text content of each cell in the same row according to the correspondence between column index and field name to form a complete material record.

[0034] In this embodiment, to improve the accuracy of field alignment, field type determination is performed on the text content of each cell during the row alignment process. Field type determination is based on the formal characteristics of the text content: text containing only numbers or decimal points is classified as a numeric field; text containing a mixture of Chinese characters and letters is classified as a name field; and text containing unit symbols is classified as a unit of measurement field. The field type determination result is cross-validated with the expected field type corresponding to the column index. When the two are inconsistent, the column classification of the cell is corrected.

[0035] Module 3 performs standardized mapping on the original name descriptions of each material record in the bill of materials to obtain standardized material names and mapping confidence.

[0036] The original name description of each material record in the bill of materials is input into the standardized mapping process. Based on a pre-built industrial material standard name library and thesaurus, the standardized mapping process performs the following processing on the original descriptions: Abbreviation expansion processing is performed on the original name descriptions, expanding abbreviations or acronyms into complete standard names based on the thesaurus; synonym unification processing is performed on the original name descriptions, mapping names with the same meaning but different expressions to the same standard name; specification parsing processing is performed on the specification parameters in the original descriptions, parsing specification expressions into structured specification parameters; and model standardization processing is performed on the model information in the original descriptions, unifying model differences from different sources to the enterprise's standard model system.

[0037] During the standardization mapping process, a multi-strategy voting method is employed to determine the mapping result. Three matching strategies are executed in parallel: string similarity matching, semantic vector similarity retrieval, and rule engine parsing. String similarity matching calculates the edit distance between the original description and each candidate name in the industrial material standard name library, outputting the candidate results with the highest similarity ranking. Semantic vector similarity retrieval encodes the original description and the names in the industrial material standard name library into semantic vectors, calculates the cosine similarity between the vectors, and outputs the candidate results with the highest similarity ranking. Rule engine parsing performs pattern matching on the original description based on predefined material naming rules, outputting candidate results that conform to the rules. Voting statistics are performed on the candidate results of the three strategies to comprehensively derive the most likely standardized mapping result and its mapping confidence.

[0038] It should be noted that the multi-strategy voting method involves aggregating the candidate results output by each of the three matching strategies, counting how many strategies recommend each candidate criterion name, and weighting the matching scores of each strategy. When a candidate criterion name is recommended by at least two strategies and has the highest weighted score, that candidate criterion name is determined as the standardized mapping result. The mapping confidence is the normalized value of the weighted score, ranging from 0 to 1.

[0039] In this embodiment, to improve the accuracy of semantic vector similarity retrieval, the semantic vector encoding employs a text encoder trained on an industrial material corpus for domain adaptation. The input layer of the text encoder receives a sequence of segmented material names, and the output layer is a fully connected layer that outputs a fixed-dimensional semantic vector. Domain adaptation training uses name pairs and their synonym annotations from an industrial material standard name database as training data, employs contrastive learning loss as the loss function, and uses the Adam optimization algorithm to update the parameters of the text encoder. The output scores of string similarity matching and rule engine parsing are respectively processed using mean normalization based on the range, scaling the scores of the three strategies to the same numerical range to ensure the comparability of scores among the strategies during the voting process.

[0040] Module 4 inputs the standardized mapping results into the industrial material knowledge graph to perform bidirectional verification and obtain the verification results.

[0041] The industrial material knowledge graph stores the standard names of materials, specification parameter systems, substitution relationships between materials, composition relationships, and the association information between materials and suppliers and historical prices.

[0042] Perform forward and reverse verification on the standardized mapping results.

[0043] Forward verification: Checks whether the standard material name in the standardized mapping result exists in the industrial material knowledge graph, and checks whether the attributes of the standard material name in the industrial material knowledge graph are complete. If the standard material name does not exist in the industrial material knowledge graph, the material corresponding to the standard material name is marked as a "new material", triggering the new material registration process.

[0044] Reverse verification: Check whether the specification parameters in the standardized mapping result fall within the standard parameter range corresponding to the standard material name in the industrial material knowledge graph. When the specification parameters exceed the standard parameter range, mark the material record as "suspected identification error" and return the material record to module 3 to re-execute the standardized mapping.

[0045] Module 4 may specifically include: Module 401 performs an entity query in the industrial material knowledge graph based on the standard material names in the standardized mapping results to determine whether the entity node corresponding to the standard material name exists. If it exists, proceed to module 402; if it does not exist, mark the material record as "new material" and generate a new material registration task.

[0046] Module 402, based on the structured specification parameters in the standardized mapping results, queries the standard parameter range associated with material entity nodes in the industrial material knowledge graph, and compares each specification parameter to see if it falls within the corresponding standard parameter range. If all specification parameters are within the standard parameter range, the verification passes; if any specification parameter exceeds the standard parameter range, the material record is marked as "suspected identification error," and the process returns to Module 3 to re-perform the standardized mapping for that material record.

[0047] It should be noted that the standard parameter range refers to the predefined reasonable value range for each specification parameter of each material in the industrial material knowledge graph. The standard parameter range is determined based on the industry standard specification data of the material and the company's historical procurement data.

[0048] In this embodiment of the application, to avoid cyclic processing caused by repeated mapping, a maximum number of remapping attempts is set for each material record. When the cumulative number of times a material record has been returned to module 3 for remapping reaches the maximum number of remapping attempts, it will no longer return to module 3, but will be directly marked as "awaiting manual review" and enter the review process of module 5.

[0049] Module 5 calculates multi-dimensional confidence scores for each material record and generates an analytical results report based on the multi-dimensional confidence scores.

[0050] For each material record verified by Module 4, a multi-dimensional confidence score is calculated. The multi-dimensional confidence score includes the following dimensions: OCR recognition confidence, layout partitioning stability, field alignment integrity, standardized mapping voting score, and knowledge graph verification pass rate.

[0051] OCR recognition confidence is the average recognition confidence of the OCR engine's output for each text field in the material record. Page layout stability is the partition confidence score of the functional area to which the material record belongs in Module 1. Field alignment integrity is the ratio of the number of successfully aligned fields in the material record to the expected total number of fields, ranging from 0 to 1. Standardized mapping voting score is the mapping confidence output by Module 3. Knowledge graph validation pass rate is the ratio of the number of validated specification parameters to the total number of specification parameters in Module 4, ranging from 0 to 1.

[0052] Of the five dimensions mentioned above, OCR recognition confidence, layout partitioning stability, and standardized mapping voting score are all normalized probability values ​​directly output by the OCR engine or layout detection model, with values ​​ranging from 0 to 1; field alignment integrity and knowledge graph verification pass rate are ratios, also ranging from 0 to 1. Therefore, the scores of the five dimensions do not require additional normalization processing and can be directly used in weighted summation calculations.

[0053] Let the scores for the five dimensions mentioned above be respectively The corresponding preset weights are respectively ,in To determine the confidence level for OCR recognition, For page layout stability, For field alignment integrity, To standardize the mapping voting score, To verify the pass rate of the knowledge graph, To correspond to the preset weights of each dimension, For dimensional indexing, The total number of dimensions is the upper bound of the summation, and it satisfies... The overall confidence score of the material record is then... for: ; Comprehensive confidence score Below the preset threshold The material records are marked as "pending review". It is preset by the user based on the actual business requirements for parsing accuracy.

[0054] For material records marked "Pending Review", a visual review interface is generated. The visual review interface highlights the questionable areas on the original document image and displays the system's reasoning process information.

[0055] In this embodiment, to continuously improve parsing accuracy, the verified correct results are fed back as positive samples to the directional information extraction process of module 2 and the standardized mapping process of module 3 for incremental learning. During incremental learning, positive samples are added to the training dataset, and parameter fine-tuning is performed on the named entity recognition model and the text encoder, respectively, updating the parameters of the named entity recognition model and the text encoder. Simultaneously, the verified standardized mapping pairs are added to the thesaurus.

[0056] Example 2 The purchasing department of a machinery manufacturing company (hereinafter referred to as "Company A") processes monthly purchase receipts from multiple suppliers, with document formats varying depending on the supplier. In Month B of a certain year, Company A received a paper purchase receipt from supplier Company B. After scanning, the document image was obtained and needed to be automatically parsed using an industrial material document parsing system to extract the material list and complete the standardization process for integration with the company's ERP system's inventory registration process. This receipt included document header information, three material detail records, a summary amount, and remarks / signatures. Some material names were written using industry abbreviations, and the specifications were expressed in a non-standard format.

[0057] The layout detection model processes the scanned image, locates the bounding boxes of each functional area in the image coordinate system, and assigns a semantic category label to each area. The model also outputs a partition confidence score for each area for reference in subsequent steps, as shown in Table 1 below.

[0058] Table 1. Results of semantic partitioning recognition on page layout All four regions had partition confidence scores higher than the partition threshold, and no region was marked as "partition uncertain". The partition results are complete and usable.

[0059] For the material details area (Area 2), table structure recognition was performed, identifying a 3-row, 6-column table structure. The columns correspond to material name, specifications, unit of measurement, quantity, unit price, and amount, respectively. During row alignment, the text in each cell was classified by field type: the first column containing Chinese characters was classified as a name field; the second column containing a mixture of letters and numbers was classified as a name field; the third column containing unit symbols was classified as a unit of measurement field; and columns 4 through 6 containing only numbers or text with decimal points were classified as numeric fields. The field type classification results were consistent with the expected column index types, eliminating the need for column classification correction.

[0060] For the document header area (Area 1), the named entity recognition model extracts entities such as supplier name, document number, and date. For the summary area (Area 3), the total amount and tax amount are extracted based on field template matching. For the remarks and signature area (Area 4), the original text is directly stored in the remarks field, as shown in Table 2 below.

[0061] Table 2 Results of Targeted Information Extraction (Original Extraction from Material Details Area) Document header area extraction results: Supplier name is "Company B", document number is "RK-B-00312", date is "Year B Month 14th". Summary area extraction results: Total amount 2255.00 yuan, tax 293.15 yuan. Remarks field: Original text: "This batch of goods is accompanied by a quality inspection report, which will take effect after the acceptance personnel sign and confirm."

[0062] Standardized mapping was performed on the original name descriptions of material lines 1 to 3. Taking "Bearing (Deep Groove Ball)" in material line 2 as an example: the candidate result of string similarity matching was "Deep Groove Ball Bearing" (normalized similarity score 0.81); the candidate result of semantic vector similarity retrieval was also "Deep Groove Ball Bearing" (normalized cosine similarity score 0.93); the rule engine parsing identified the candidate result "Deep Groove Ball Bearing" based on the naming rule "[Structure Type] + Bearing". All three strategies recommended the same candidate name, and after voting statistics, the standard name was determined to be "Deep Groove Ball Bearing", and the mapping confidence was the result of normalizing the weighted scores of the three strategies. The specification parsing resolved "6208-2RS" into structured parameters: inner diameter 40 mm, outer diameter 80 mm, width 18 mm, and sealing type double-sided rubber seal, as shown in Table 3 below.

[0063] Table 3 Standardized mapping results The original description of material line 3, "hydraulic cylinder seal ring assembly", showed conflicting recommendations among the three strategies. The rule engine score was relatively low, and the candidate name "hydraulic seal ring assembly" with the highest weighted score was ultimately used as the mapping result. The mapping confidence was 0.83, which was lower than the first two records.

[0064] Perform steps 401 and 402 sequentially on the three material records.

[0065] Step 401: Search for the three entity nodes "hexagonal head bolt", "deep groove ball bearing" and "hydraulic seal ring assembly" in the industrial material knowledge graph. All three exist in the graph and there is no "new material" mark. Proceed to step 402.

[0066] Step 402: Compare the structured specifications of each material record with the standard parameter ranges of the corresponding entity nodes in the diagram. Material line 1's specifications M12×50 and galvanized surface treatment are both within the standard parameter range; Material line 2's inner diameter 40 mm, outer diameter 80 mm, and width 18 mm are all within the standard parameter range; Material line 3's inner diameter 50 mm, outer diameter 65 mm, thickness 8 mm, and material nitrile rubber are all within the standard parameter range. All three records passed the verification, as shown in Table 4 below.

[0067] Table 4. Results of bidirectional verification of knowledge graph Calculate the overall confidence score for each of the three material records. ,in , To determine the confidence level for OCR recognition, For page layout stability, For field alignment integrity, To standardize the mapping voting score, This is to verify the pass rate of the knowledge graph. The preset weights for each dimension in this analysis are as follows: , , , , ,satisfy Preset threshold Company A sets the accuracy requirement for the warehouse registration business to 0.80, as shown in Table 5 below.

[0068] Table 5. Multidimensional Confidence Scores and Analysis Conclusions The overall confidence score of all three material records is higher than the threshold. Records with no entries are marked as "pending review." The system generates a parsing result report and pushes the structured bill of materials to Company A's ERP system to complete the warehousing registration.

[0069] Throughout the analysis process, the data starts with scanned images of documents. Step 1 obtains the coordinates of functional areas with semantic labels. Step 2 extracts raw fields from each area and aggregates them into a bill of materials. Step 3 converts the original names and descriptions into standardized names using a three-strategy voting process, along with mapped confidence levels. Step 4 uses a knowledge graph to perform bidirectional verification of the standardized results and outputs the verification pass rate. Step 5 aggregates the five-dimensional scores generated in the first four steps to calculate a comprehensive confidence score and generate the final report. The output of each step directly serves as the input for the next step, and the five-dimensional scores... Originating from step 1 Originating from step 2 Originating from step 3 Originating from step 4, it demonstrates the complete logical link of data flow between each step.

[0070] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0071] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal document parsing system for intelligent identification of industrial materials, characterized in that, include: The layout partitioning recognition module is used to perform semantic partitioning recognition on industrial material document images to obtain the functional areas and their semantic category labels; The targeted information extraction module is used to extract targeted information for each functional area based on the semantic category labels of each functional area, and obtain a structured bill of materials. The standardization mapping module is used to perform standardization mapping on the original name descriptions of each material record in the bill of materials. It uses a multi-strategy voting method to determine the mapping result and obtains the standardized material name and mapping confidence. The knowledge graph verification module is used to input the standardized mapping results into the industrial material knowledge graph to perform bidirectional verification and obtain the verification results; The confidence assessment module is used to calculate multi-dimensional confidence scores for each material record and generate an analytical result report based on the multi-dimensional confidence scores.

2. The multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, The semantic partitioning recognition of the layout is achieved through the layout detection model. The input layer of the layout detection model receives the image feature sequence of the document image and the corresponding text position encoding sequence. The output layer contains two parallel fully connected layers. One fully connected layer outputs the bounding box coordinates of each region, and the other fully connected layer outputs the semantic category label of each region. The partitioning confidence score is calculated for each functional region output by the layout detection model. Regions with partitioning confidence scores lower than the partitioning threshold are marked as partitioning uncertain.

3. The multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, Targeted information extraction is performed on each functional area, including: For the material details area, the table structure is identified to obtain the row and column structure of the table. Based on the row and column structure, row alignment processing is performed on each row of material records. Each row of material is decomposed into the fields of material name, specification model, unit of measurement, quantity, unit price, and amount. The row alignment processing uses prior knowledge of the layout as a constraint condition. For the document header area, a named entity recognition model is used to extract the supplier name, document number, and date information; For the summary area, template matching is performed based on predefined field templates to extract the total amount and tax amount numerical fields; For the remarks and signature areas, extract the text content as an additional explanation field and store it in the remarks field of the bill of materials.

4. A multimodal document parsing system for intelligent identification of industrial materials according to claim 3, characterized in that, During row alignment, the text content of each cell is classified into numeric fields, text containing only numbers or decimals is classified into name fields, and text containing unit symbols is classified into unit fields. The field type classification results are cross-validated with the expected field type corresponding to the column index. If the two are inconsistent, the column classification of the cell is corrected.

5. A multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, Standardization mapping includes: performing abbreviation expansion processing on the original name description, expanding the abbreviation or acronym into a complete standard name based on a thesaurus; performing synonym unification processing on the original name description, mapping names with the same meaning but different expressions to the same standard name; performing specification parsing processing on the specification parameters in the original description, parsing the specification expressions into structured specification parameters; and performing model standardization processing on the model information in the original description, unifying model differences from different sources into the enterprise standard model system.

6. A multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, In the multi-strategy voting method, when a candidate standard name is recommended by at least two strategies and has the highest weighted score, the candidate standard name is determined as the standardized mapping result; the mapping confidence is the value after normalization of the weighted score, and the value ranges from 0 to 1; the text encoder in the semantic vector similarity retrieval is a text encoder trained on industrial material corpus for domain adaptation, and the parameters are updated using contrastive learning loss as the loss function.

7. A multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, Forward verification includes: checking whether the standard material name in the standardized mapping result exists in the industrial material knowledge graph. If it does not exist, the material record is marked as a new material and the new material registration process is triggered. Reverse verification includes: checking whether the specification parameters in the standardized mapping result fall within the range of standard parameters corresponding to the standard material name in the industrial material knowledge graph. When the specification parameters exceed the range of standard parameters, the material record is marked as a suspected identification error and the standardized mapping is re-executed.

8. A multimodal document parsing system for intelligent identification of industrial materials according to claim 7, characterized in that, Set a maximum number of remapping attempts for each material record. When the cumulative number of times a material record has been re-executed for standardized mapping reaches the maximum number of remapping attempts, the material record is directly marked as pending manual review.

9. A multimodal document parsing system for intelligent identification of industrial materials according to claim 1, characterized in that, Multi-dimensional confidence scoring includes OCR recognition confidence. Page layout stability Field alignment integrity Standardized mapping voting score Knowledge graph validation pass rate Overall confidence score ,in Preset weights for each dimension, It is a dimension index and satisfies ; The overall confidence score Below the preset threshold Material records are marked as pending review. Correct results after review and confirmation are used as positive samples and fed back to the targeted information extraction process and standardized mapping process for incremental learning.

Images