Multimodal-based engineering configuration table content recognition method and system

Abstract

The present application belongs to the field of computer vision and intelligent manufacturing technology, and specifically relates to a kind of engineering configuration table content identification method and system based on multi-modal, comprising the following specific steps: S1, at least one engineering configuration table image is acquired, and identification task is established;S2, the engineering configuration table image is subjected to adaptive image enhancement and layout geometry reconstruction, and regular table image is obtained;S3, the regular table image is executed multi-modal content identification, to output the table structure information and text content information of engineering configuration table;S4, based on the table structure information and text content information, reconstruct structured data table, at least including reconstruction row and column index and merging relationship, and the same kind of accessory is executed summary and quantity statistics;S5, the structured data table is executed field check and standardization processing, and after passing the check, the standardized result is output.The present application realizes the accurate identification and standardization arrangement of engineering material field in table under various complex conditions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent manufacturing technology, specifically relating to a method and system for recognizing the content of an engineering configuration table based on multimodality. Background Technology

[0002] Engineering configuration sheets are fundamental technical documents widely used in engineering fields such as electrical assembly, mechanical equipment manufacturing, and industrial installation and commissioning. Their content typically includes material information such as component names, model specifications, units, and quantities. With the increasing digitization of engineering projects, enterprises generally hope to automatically convert engineering configuration sheets in PDF, paper, or photographic formats into structured data to meet subsequent business needs such as material statistics, automatic verification, automatic order placement, and integration with ERP systems. Therefore, content recognition of engineering configuration sheets has become an important technological direction in the fields of intelligent manufacturing and visual information processing.

[0003] In existing technologies, the parsing of engineering configuration table images typically employs a combination of optical character recognition (OCR) and traditional table recognition methods. A typical process includes: preprocessing the captured image (such as denoising, enhancement, and skew correction), table line detection, cell segmentation, text box extraction and OCR character recognition, and rule-based field attribution inference, in order to achieve the conversion from image to table data.

[0004] However, in practical applications of engineering configuration tables, the aforementioned table recognition methods based on rules or single modal features have significant limitations. First, engineering configuration tables are often captured by mobile phones or handheld devices, resulting in image quality issues such as tilt, distortion, blurring, and uneven lighting. This makes it difficult for traditional table line detection and cell segmentation algorithms to accurately identify the table structure, leading to incomplete or misclassified cell boundaries. Second, engineering configuration tables contain numerous horizontally or vertically merged cells, and the row and column division methods differ significantly across table templates. Traditional table parsing algorithms based on line detection or contour detection struggle to handle complex merge relationships, easily resulting in structure reconstruction failures. Third, textual content such as engineering material names, multi-level model specifications, and engineering equipment terminology has strong industry characteristics and inconsistent expression methods. OCR output text often faces difficulties in field classification and confusion regarding field attribution. Furthermore, the combination of OCR misidentification and complex layouts further exacerbates ambiguity and leads to error propagation.

[0005] Engineering applications require more than just "reading text" for configuration table recognition; they also emphasize "correct structure, correct field attribution, correct restoration of merged cells, and output that can be directly used in engineering management systems." For example, the automatic structuring of engineering configuration tables typically requires accurately restoring the row and column topology and merging relationships at the table structure level, and stably extracting key fields such as accessory names, model specifications, units, and quantities at the field level. Furthermore, to reduce the risk of structural conflicts and misalignments in complex scenarios, it is necessary to constrain the structural results and resolve conflicts, and to perform engineering-domain consistency verification and standardization on the field results to improve the quality and usability of structured data.

[0006] Therefore, there is an urgent need for an image structure recognition technology for engineering configuration tables that can comprehensively utilize the visual structure information, text content information, and spatial layout information of tables, while taking into account the semantic constraints of the engineering field, under conditions such as fluctuating photo quality, significant template differences, complex merged cells, and diverse engineering terminology. This technology aims to improve the robustness and accuracy of table structure reconstruction and field extraction, and meet the requirements of engineering material management systems for data accuracy and consistency. Summary of the Invention

[0007] Based on the above description, the present invention provides a method and system for identifying the content of engineering configuration tables based on multimodality, so as to achieve accurate identification and standardized organization of engineering material fields in tables under various complex conditions.

[0008] On the one hand, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a method for recognizing the content of an engineering configuration table based on multimodality, comprising the following specific steps:

[0009] S1. Obtain at least one engineering configuration table image and establish a recognition task for the engineering configuration table image;

[0010] S2. Perform adaptive image enhancement and layout geometric reconstruction on the project configuration table image to obtain a regular table image. The adaptive image enhancement and layout geometric reconstruction include at least: performing image quality assessment on the project configuration table image to obtain quality features; determining an enhancement strategy based on the quality features and performing image enhancement on the project configuration table image; performing table region detection and extracting table key points on the enhanced image; performing perspective transformation on the table region based on the table key points to perform geometric correction; and performing edge defect repair and local structure completion on the geometrically corrected image.

[0011] S3. Perform multimodal content recognition on the regularized table image to output the table structure information and text content information of the engineering configuration table. The multimodal content recognition includes: acquiring the image features and text features of the regularized table image, and inputting them into a multimodal recognition model to jointly infer the row and column structure, the relationship between merged cells, and the content of preset fields; wherein, the preset fields include at least the accessory name, model specification, unit and quantity, and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells.

[0012] S4. Reconstruct a structured data table based on the table structure information and text content information, including at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories;

[0013] S5. Perform field validation and normalization on the structured data table, and output the normalization result after the validation passes. The normalization result includes at least an editable table file and / or structured data interface data for import into the business system.

[0014] This method addresses common problems in engineering configuration tables encountered during scanning / photography, such as blurring, noise, shadow occlusion, skewed perspective, watermark interference, and missing table borders. First, it employs quality assessment-driven adaptive enhancement and geometric reconstruction to detect table areas, correct key point perspective, and repair missing parts, resulting in a well-structured and structurally closed table image. This reduces the cumulative errors caused by structural breaks and misidentification of text at the source. Further, multimodal joint encoding aligns visual grids with text semantics, jointly inferring row and column structures, merged cell relationships, and fields such as accessory names, model specifications, units, and quantities. When merged cells or fields are missing, they are filled in based on adjacent context, significantly improving field integrity and consistency in complex header and merged cell scenarios. Subsequently, a structured data table is reconstructed, and similar accessories are automatically summarized and statistically analyzed. This achieves end-to-end automated output from images to editable tables / structured interface data. Combined with field validation and standardization, this reduces the error rate of key fields such as units, specifications, and quantities, decreasing manual entry and verification costs and improving the efficiency and reliability of engineering configuration data entry.

[0015] Based on the above technical solution, the present invention can be further improved as follows.

[0016] Furthermore, the image quality assessment in S2 quantifies the quality of the engineering configuration table image into a quality feature vector, which includes at least sharpness, noise, brightness / contrast, shadow occlusion, tilt, and watermark interference indicators; wherein the sharpness indicator is represented by the Laplacian operator response variance. The shadow occlusion index is expressed as the percentage of shadow area: ;

[0017] in, The image grayscale matrix, For the Laplace operator, The area of ​​the shaded region. The area of ​​the table region; and, based on the quality feature vector and the preset threshold set. Adaptive selection of enhancement strategies, including at least: when Deblurring / sharpening is performed when the noise level is below a certain threshold. Denoising is performed during operation; when brightness / contrast deviates from the threshold range, illumination equalization and adaptive contrast stretching are performed. Shadow compensation is applied when the tilt index exceeds [a certain threshold]. Time-enhanced geometric correction priority, when watermark interference index exceeds Watermark suppression and background suppression are performed simultaneously.

[0018] The above technical solution quantifies the image quality of the engineering configuration table into feature vectors that include sharpness, noise, brightness / contrast, shadow occlusion, tilt, and watermark interference. Combined with threshold set adaptive selection of strategies such as denoising, sharpening, illumination equalization, shadow compensation, and watermark suppression, it can "enhance as needed" for different acquisition conditions, avoiding over-enhancement or under-enhancement caused by uniform processing. This improves the separability of table lines and text, reduces OCR misidentification and structural breakage rate, and enhances the robustness of subsequent geometric correction and multimodal recognition.

[0019] Furthermore, in S2, the table region detection outputs the detection confidence score of the table candidate region, and selects the candidate region with the highest confidence score as the table region; the table key points include at least the corner points of the table outer border and / or grid intersections; when performing perspective transformation based on the table key points, the homography matrix is ​​estimated. Achieve geometric correction, and the This is obtained by minimizing the reprojection error: ;

[0020] in, Key points before calibration For the corresponding point on the target plane, For homogeneous normalized projection; and, the The solution employs robust estimation to remove outlier keypoints, and the robust estimation includes at least threshold-based methods. Interior point determination and interior point scaling constraints .

[0021] By using the above technical solutions, the confidence scores of candidate table regions are output and the region with the highest confidence score is selected, which can reduce the structural inference bias caused by the inadvertent inclusion of non-table regions. The homography matrix is ​​estimated by using the corner points / grid intersections of the outer frame and the reprojection error is minimized, which can achieve accurate correction of perspective and tilt distortion in photographs. At the same time, a robust estimation based on the threshold τ and the ratio of interior points ρ is introduced to remove abnormal key points, which can significantly improve the stability and consistency of geometric correction and reduce systematic errors in subsequent row and column segmentation and merging relationship judgment.

[0022] Furthermore, the edge defect repair and local structure completion in S2 include: performing connectivity analysis on the geometrically corrected set of line segments to locate the broken endpoints, and extending the broken endpoints to reconstruct the missing line segments; wherein, the line extension adopts straight line fitting and constrains the straightness of the lines with maximum deviation: ;

[0023] in, To fit a straight line, To extend the line pixels or sampling points within the region, To ensure linearity tolerance, structural constraints are introduced during the completion process to guarantee mesh consistency. These structural constraints include at least the following: the angle difference between adjacent horizontal line segments is no greater than [value missing]. Parallel consistency, the difference between the horizontal and vertical principal directions is no greater than The orthogonality consistency and cell closure check are performed; when the closure check fails, the missing area is iteratively filled until the structural constraints are met or the maximum number of iterations is reached.

[0024] The above technical solution uses connectivity analysis to locate broken endpoints in the geometrically corrected line segment set, and then uses straight line fitting to extend and reconstruct the missing line segments, which can effectively repair common problems in engineering sites such as missing borders and broken meshes. The maximum deviation constrains the straightness to suppress extension offset and misconnection. At the same time, parallel consistency, orthogonal consistency and cell closure checks are introduced, and iterative completion is performed when the cells are not closed, which can ensure the consistency of mesh topology and the complete closure of cells, thereby reducing cell splitting / merging errors and improving the reliability of structured reconstruction.

[0025] Furthermore, the multimodal recognition model in S3 includes a visual encoder, a text encoder, and a cross-modal fusion module. The visual encoder outputs visual features of the table grid and layout, the text encoder outputs semantic features from the OCR text sequence, and the cross-modal fusion module performs joint alignment of visual and text features based on a cross-attention mechanism to simultaneously infer row and column structure, merged cell relationships, and preset field content. Additionally, it outputs a field confidence score for each preset field, where the field confidence score is determined by the maximum value of the field classification probability. ;

[0026] in, For fields The corresponding model outputs logits; when merged cells or missing fields are detected, the context of adjacent cells under the merge relationship constraint is used as the condition input for field completion, and the confidence of the completion results is filtered for credibility using the field confidence.

[0027] The above technical solution employs a combination of visual encoder, text encoder, and cross-modal fusion (cross-attention) to achieve joint alignment of visual grids and OCR semantics. This enables simultaneous inference of structure and fields in scenarios such as complex headers, merged cells, and misaligned multi-row and multi-column structures, reducing mismatches caused by relying solely on a single modality. The confidence score of fields is output and used for filtering the completion results. This can suppress unreliable inferences when completing missing fields or merged cells, improving field completeness while controlling the risk of erroneous completion. This makes the output results more interpretable and more suitable for automatic data entry in engineering projects.

[0028] Furthermore, the field validation and normalization process in S5 constructs a field consistency score, and the output is determined based on the consistency score and a threshold; wherein, the consistency score is obtained by weighting the results of multiple rule validations: ;

[0029] in, Indicates the first Whether the rule is approved To correspond to the weights, the rules must include at least: dictionary normalization and synonym mapping rules for part names, format template matching and key parameter extraction rules for model specifications, unit mapping and unified conversion rules, and numerical validity rules for quantities; when or confidence level of any field A review marker is generated in time, and the review marker and the corresponding field position index are carried when the editable table file and / or structured data interface data are output.

[0030] The above technical solution constructs a weighted consistency score for key fields such as name, specifications, unit, and quantity. A rule base is used to achieve dictionary normalization, template matching, unit conversion, and numerical validity verification. Before output, the structured results can be corrected and standardized under domain constraints, significantly reducing the risk of high-risk errors such as "unit confusion, inconsistent specification formats, and abnormal quantities" entering the business system. When the score or confidence level is insufficient, a review mark is generated with a field position index, which can achieve accurate positioning and efficient manual review, reduce the cost of full table verification, and improve the reliability and traceability of data import.

[0031] Secondly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: A multimodal engineering configuration table content recognition system, comprising: a task management module, used to acquire at least one engineering configuration table image and establish a recognition task, generate a task identifier, and record task parameters and processing status; an adaptive image enhancement and layout geometry reconstruction module, used to perform image quality assessment on the engineering configuration table image to obtain quality features, and select an enhancement strategy to enhance the image based on the quality features; perform table region detection on the enhanced image and extract table key points; perform perspective transformation on the table region based on the table key points to perform geometric correction, and perform edge defect repair and local structure completion on the geometrically corrected image to output a regular table image; and a multimodal content recognition module, used to acquire image features and text features of the regular table image, and input multiple... The modal recognition model jointly infers the table's row and column structure, merged cell relationships, and preset field content. The preset fields include at least the accessory name, model specifications, unit, and quantity. When merged cells or missing fields exist, the missing fields are filled in based on the context of adjacent cells. A structured reconstruction and summary statistics module is used to reconstruct the structured data table based on the row and column structure, merged cell relationships, and preset field content. This includes at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories. A field validation and normalization module is used to perform field validation and normalization processing on the structured data table, and generate standardized results after successful validation. An output interface module is used to output the standardized results, which include at least an editable table file and / or structured data interface data for import into business systems.

[0032] Through the above technical solution, the system integrates a pipeline of "task management—adaptive enhancement and geometric reconstruction—multimodal recognition—structured reconstruction statistics—verification and standardization—interface output" to achieve end-to-end automated processing of engineering configuration tables from image input to editable table / structured interface data. By standardizing table images and performing structured reconstruction, structural errors caused by tilting and missing parts are reduced; by verifying and standardizing and unifying the output interface, data consistency and project usability are improved, significantly reducing manual data entry and aggregation costs.

[0033] Furthermore, the image quality assessment in the adaptive image enhancement and layout geometry reconstruction module includes at least the following verifiable quality items: sharpness, noise level, brightness / contrast shift, shadow / occlusion ratio, tilt angle, and watermark / background interference intensity; and configures a corresponding threshold or grading rule for each quality item to adaptively select an enhancement strategy accordingly; the enhancement strategy includes at least one or more combinations of denoising, sharpening or deblurring, adaptive adjustment of brightness and contrast, shadow suppression, background suppression, and / or watermark suppression, and supports configuring the strategy priority according to task parameters.

[0034] The above technical solution subdivides image quality into actionable indicators such as sharpness, noise, brightness / contrast, shadow occlusion, tilt, and watermark interference. It also configures threshold / grading rules to drive the adaptive combination and priority adjustment of enhancement strategies, so that enhancement can be performed "on demand" under different acquisition conditions. This avoids over-processing that introduces artifacts or under-processing that causes information to become inseparable, thereby improving the readability of table lines and text and reducing errors in subsequent detection, OCR, and structural inference.

[0035] Furthermore, the table region detection at least uses object detection or semantic segmentation to output the table's bounding region, and performs morphological screening and connected component filtering on the detection results to eliminate non-table regions; the table key points at least include the corner points of the table's outer border and / or grid intersections, and key point extraction at least uses line segment detection and intersection solving or prediction methods based on key point networks; the perspective transformation at least uses geometric correction parameters obtained based on four corner points or multi-point fitting, and robust sampling and consistency verification are used to improve correction stability when key points are abnormal; the edge defect repair and local structure completion at least include broken line segment endpoint localization, line segment extension, intersection reconstruction, and cell closure check, and when the closure check fails, iterative completion or regression to a backup reconstruction strategy is triggered.

[0036] The above technical solutions employ object detection / semantic segmentation combined with morphological screening and connected component filtering to suppress interference from non-table regions and improve the accuracy of table localization. By using the corner points / grid intersections of the outer frame as key points and supporting line segment intersection methods or key point network prediction, along with perspective correction through four-corner point / multi-point fitting, robust sampling and consistency verification are introduced when key points are abnormal, which can significantly enhance the stability of geometric correction. Through fracture endpoint localization, line segment extension, intersection reconstruction, closure checks, and iteration / backoff strategies, grid closure and topological consistency are ensured, reducing cell splitting / merging errors.

[0037] Furthermore, the multimodal content recognition module includes a visual encoding unit, a text recognition unit, and a cross-modal fusion inference unit:

[0038] Visual encoding units are used to extract grid lines, cell boundaries, and layout features;

[0039] The text recognition unit is used to perform text recognition on a regular table image and output the text content and its location box;

[0040] The cross-modal fusion inference unit is used to align text location boxes and visual grid features, jointly output row and column structure, merged cell relationship, and field values ​​of accessory name, model specification, unit and quantity, and output confidence scores for the field values;

[0041] Specifically, missing field completion is based on inference and completion based on the coverage of adjacent cells in the same row / column, the same merged cell, and field template constraints. The field validation and normalization module processes the part name synonyms, model specification format templates, unit mapping and conversion, and quantity legality checks based on the dictionary and rule base, and generates a review mark when the confidence level is lower than the threshold or the validation fails. The output interface module carries the review mark and its corresponding row and column position index when outputting editable table files and / or structured data interface data.

[0042] By employing the aforementioned technical solutions, the combined output of row and column structure, merging relationships, and accessory name / specification / unit / quantity fields, along with confidence levels, using "visual encoding + text recognition + cross-modal alignment fusion," can improve the accuracy of structure-field alignment in scenarios with complex headers and merged cells. Missing fields are filled based on row and column neighborhood, merging coverage, and template constraints, improving field completeness and controllability. Furthermore, by combining dictionary / rule base verification standardization, a review mark is generated when the confidence level is low or the verification fails. The output carries the review mark and row and column position index, enabling traceable, locatable, efficient manual review and reliable warehousing.

[0043] Compared with the prior art, the technical solution of this application has the following beneficial technical effects:

[0044] 1. To address common issues in engineering configuration tables during scanning / photography, such as blurring, noise, uneven brightness, shadow occlusion, tilted perspective, watermark interference, and missing edges, the system / method first performs a quality assessment and adaptively selects an enhancement strategy accordingly. Then, it completes table region detection, key point extraction, and robust perspective correction. Combined with line segment connectivity analysis, extension repair, and iterative completion of structural constraints, it obtains a table image with a closed structure and regular geometry. This reduces the accumulation of errors caused by structural breaks and misalignments from the source, improving the stability and accuracy of subsequent structure and field recognition.

[0045] 2. By aligning visual grids and OCR semantics through a multimodal model, the system jointly infers row and column structures, merged cell relationships, and fields such as part name / model specifications / unit / quantity. In scenarios involving merging or missing parts, it utilizes adjacent contexts for reliable completion. Subsequently, it reconstructs structured data tables and summarizes statistics for similar parts. Then, it uses dictionaries and rule bases for field validation, unit conversion, and normalization. Combined with consistency scores / confidence levels, it generates review markers carrying row and column position indexes. Finally, it outputs editable table files and / or structured interface data, achieving end-to-end automated, low-error, and traceable business system import, significantly reducing manual data entry and review costs. Attached Figure Description

[0046] Figure 1This is a flowchart illustrating the overall process of the multimodal engineering configuration table content recognition method according to Embodiment 1 of the present invention.

[0047] Figure 2 This is a flowchart of the adaptive image enhancement and layout geometry reconstruction method of Embodiment 1 of the present invention;

[0048] Figure 3 This is a schematic diagram of the multimodal content recognition model structure and joint inference of the method in Embodiment 1 of the present invention;

[0049] Figure 4 This is a flowchart illustrating the structured data table reconstruction and summary statistics of similar parts according to Embodiment 1 of the present invention.

[0050] Figure 5 This is a flowchart illustrating the field validation, normalization, and output process of the method in Embodiment 1 of the present invention.

[0051] Figure 6 This is a schematic diagram of the overall system architecture and operating environment of the system in Embodiment 2 of the present invention;

[0052] Figure 7 This is a schematic diagram of the module call timing of the system in Embodiment 2 of the present invention;

[0053] Figure 8 This is a logic diagram of the state machine and exception retry of the task management module in Embodiment 2 of the present invention;

[0054] Figure 9 This is a block diagram of the adaptive image enhancement and layout geometry reconstruction module of the system in Embodiment 2 of the present invention;

[0055] Figure 10 This is a schematic diagram of the end-to-end closed loop of the system in Embodiment 2 of the present invention. Detailed Implementation

[0056] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0058] In this embodiment, "engineering configuration table image" refers to a document image containing table structure and parts list fields, and the source includes, but is not limited to, scanned images, mobile phone photos, screenshots, or exported images; "normalized table image" refers to a table image that has undergone image enhancement, table area cropping and perspective correction, edge defect repair, and local structure completion, so that the main direction of the table is approximately horizontal / vertical, the boundaries are closed, and the structure is easy to identify; "preset fields" include at least the parts name, model specifications, unit, and quantity.

[0059] Example 1:

[0060] refer to Figures 1-5 A method for recognizing the content of an engineering configuration table based on multimodality includes the following specific steps:

[0061] S1. Obtain at least one engineering configuration table image and establish a recognition task for the engineering configuration table image;

[0062] S2. Perform adaptive image enhancement and layout geometric reconstruction on the project configuration table image to obtain a regular table image. The adaptive image enhancement and layout geometric reconstruction include at least: performing image quality assessment on the project configuration table image to obtain quality features; determining an enhancement strategy based on the quality features and performing image enhancement on the project configuration table image; performing table region detection and extracting table key points on the enhanced image; performing perspective transformation on the table region based on the table key points to perform geometric correction; and performing edge defect repair and local structure completion on the geometrically corrected image.

[0063] S3. Perform multimodal content recognition on the regularized table image to output the table structure information and text content information of the engineering configuration table. The multimodal content recognition includes: acquiring the image features and text features of the regularized table image, and inputting them into a multimodal recognition model to jointly infer the row and column structure, the relationship between merged cells, and the content of preset fields; wherein, the preset fields include at least the accessory name, model specification, unit and quantity, and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells.

[0064] S4. Reconstruct a structured data table based on the table structure information and text content information, including at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories;

[0065] S5. Perform field validation and normalization on the structured data table, and output the normalization result after the validation passes. The normalization result includes at least an editable table file and / or structured data interface data for import into the business system.

[0066] Furthermore, for S1-1 image acquisition, the system acquires the engineering configuration table image I0 from any of the following sources: JPEG / HEIC images captured by the terminal camera; TIFF / PNG images output by the scanner; images exported from the document management system or uploaded by the business system.

[0067] To ensure compatibility across multiple scales, this embodiment scales the shorter side of the input image proportionally to the range of 1024~2048 pixels (1600 pixels in this example), while preserving the original resolution for future coordinate write-back.

[0068] S1-2 Establish a recognition task. Create a recognition task record task_id for each image, which should include at least the following: task_id: unique identifier; source_type: scan / take a picture / export; timestamp: acquisition time; output_mode: XLSX / JSON / both; retry_limit: maximum number of retries (2 in the example); quality_profile: scene parameter configuration (scanning scene or taking a picture scene).

[0069] Output: task_id and image I0.

[0070] Preferably, the image quality assessment in S2 quantifies the quality of the engineering configuration table image into a quality feature vector, including at least sharpness, noise, brightness / contrast, shadow occlusion, tilt, and watermark interference indicators; wherein the sharpness indicator is represented by the Laplacian operator response variance. The shadow occlusion index is expressed as the percentage of shadow area: ;

[0071] in, The image grayscale matrix, For the Laplace operator, The area of ​​the shaded region. The area of ​​the table region; and, based on the quality feature vector and the preset threshold set. Adaptive selection of enhancement strategies, including at least: when Deblurring / sharpening is performed when the noise level is below a certain threshold. Denoising is performed during operation; when brightness / contrast deviates from the threshold range, illumination equalization and adaptive contrast stretching are performed. Shadow compensation is applied when the tilt index exceeds [a certain threshold]. Time-enhanced geometric correction priority, when watermark interference index exceeds Watermark suppression and background suppression are performed simultaneously.

[0072] Preferably, in S2, the table region detection outputs the detection confidence score of the table candidate regions, and selects the candidate region with the highest confidence score as the table region; the table key points include at least the corner points of the table outer border and / or grid intersections; when performing perspective transformation based on the table key points, the homography matrix is ​​estimated. Achieve geometric correction, and the This is obtained by minimizing the reprojection error: ;

[0073] in, Key points before calibration For the corresponding point on the target plane, For homogeneous normalized projection; and, the The solution employs robust estimation to remove outlier keypoints, and the robust estimation includes at least threshold-based methods. Interior point determination and interior point scaling constraints .

[0074] Preferably, the edge defect repair and local structure completion in S2 include: performing connectivity analysis on the geometrically corrected set of line segments to locate the broken endpoints, and extending the broken endpoints to reconstruct the missing line segments; wherein, the line extension adopts straight line fitting and constrains the straightness of the lines with maximum deviation: ;

[0075] in, To fit a straight line, To extend the line pixels or sampling points within the region, To ensure linearity tolerance, structural constraints are introduced during the completion process to guarantee mesh consistency. These structural constraints include at least the following: the angle difference between adjacent horizontal line segments is no greater than [value missing]. Parallel consistency, the difference between the horizontal and vertical principal directions is no greater than The orthogonality consistency and cell closure check are performed; when the closure check fails, the missing area is iteratively filled until the structural constraints are met or the maximum number of iterations is reached.

[0076] Furthermore, the S2-1 image quality is assessed and quantized into a quality feature vector. A quality assessment is then performed on the engineering configuration table image I0 to obtain the quality feature vector: ;

[0077] These include at least the clarity index, noise index, brightness / contrast index, shadow occlusion index, tilt index, and watermark interference index.

[0078] The sharpness index is represented by the variance of the Laplace operator response: Where I is the gray-level matrix, For the Laplace operator (the example uses a 3×3 Laplace kernel).

[0079] Shadow occlusion is expressed as the percentage of shadow area: ;in The area of ​​the shaded region. This represents the area of ​​the table region; if the table region has not yet been determined, candidate table masks can be obtained by coarsely locating the "text / line-dense area". Then, the shadow mask is segmented using a luminance channel (such as the Y channel or V channel) threshold. ,by Approximate calculation; after completing the S2-3 table region detection, it can be corrected using a mask of the actual table region. .

[0080] Noise Index High-frequency energy and smoothed residuals can be used for estimation (example: smoothing an image with a small window and using the variance of the residuals as noise intensity).

[0081] Brightness / contrast index The mean brightness, standard deviation, and histogram skewness can be used to measure the brightness. When the mean brightness deviates from the expected range or the contrast is too low, it is determined that illumination equalization and contrast stretching are needed.

[0082] Inclined index The angle can be estimated from the main direction angle in the table: perform Hough line detection on the edge map and count the deviation of the main peak angle from the horizontal / vertical direction as the inclination; or it can be obtained by fitting subsequent key points.

[0083] Watermark Interference Index It can be detected by "low opacity large area repeating texture" or "regular diagonal pattern / character watermark" (example: joint discrimination of frequency domain energy concentration + text repeating connected component density).

[0084] S2-2 Adaptively selects and executes enhancement strategies based on a threshold set, where a preset threshold set is defined as follows: Example values ​​(configurable): (Resolution threshold, unit depends on operator implementation); (Noise threshold); The corresponding average brightness range is [80, 180], and the lower limit of contrast ratio σ is ≥35. (Shadow proportion threshold); (Tilt threshold); (Watermark interference intensity threshold, after normalization).

[0085] Adaptive enhancement is performed based on the quality feature vector and threshold set, including at least the following strategy selection logic: when When the noise level is above the threshold (or equivalently, "below acceptable quality"), perform deblurring / sharpening (example: USM unsharpening mask, radius 1.2, intensity 0.6; or use a lightweight deblurring network); when the noise level is above the threshold (or equivalently, "below acceptable quality"), perform denoising (example: non-local mean or bilateral filtering; search window 21, template window 7); when the brightness / contrast deviates from the threshold range, perform illumination equalization and adaptive contrast stretching (example: CLAHE, mesh 8×8, clipping threshold 2.0); when... When the skew index exceeds a certain threshold, perform shadow compensation (example: estimate the background illumination surface based on guided filtering and perform division correction; or Retinex-like methods); when the skew index exceeds a certain threshold... When the watermark interference index exceeds a certain threshold, increase the priority of geometric correction (i.e., adopt stricter outlier rejection and higher iteration budget in subsequent keypoint screening and robust estimation); when the watermark interference index exceeds a certain threshold... At that time, watermark suppression and background suppression are performed (example: frequency domain notch filtering + morphological suppression; or watermark separation network to obtain watermark layer and subtract).

[0086] Output enhanced image And record an "enhancement strategy log" (for review and interpretability), for example: enh_log={sharpen:on, denoise:on, clahe:on, shadow:on, watermark:off}.

[0087] S2-3 Table region detection and selection of candidate regions with the highest confidence. Perform table region detection and output a set of candidate regions. and its detection confidence This embodiment can be implemented using any feasible method: Depth detection: Lightweight object detection network outputs a table frame; Rule detection: Joint localization using connected components + line segment density + text density.

[0088] Select the region with the highest confidence level from the candidate regions as the table region: Perform clipping / boundary expansion on b* (example: expand outwards by 2%~5% to avoid the boundary line being clipped) to obtain the table area image. .

[0089] Exceptional branch: If (Example) If the result is negative, then "degenerate localization" is triggered: the entire map is used as the candidate table region or "the largest dense rectangle region of line segments" is used to replace b*, and the task is marked as "requires review - table region is uncertain".

[0090] S2-4 Extraction of table key points. Table key points must include at least the corner points of the table's outer border and / or grid intersections. The specific implementation is as follows:

[0091] Line segment detection and principal direction estimation: To perform edge detection, a probabilistic Hough transform is executed to obtain a set of horizontal / vertical line segments. The main direction angle is statistically analyzed to obtain the initial tilt angle estimate θ0; outer frame corner points: the four corner points p1..p4 are obtained based on the intersection of the outermost horizontal line and the vertical line; if the outer frame line is incomplete, the outer boundary line is fitted with RANSAC and the intersection points are calculated; grid intersection points: the internal horizontal / vertical line segments are clustered (by position / spacing), the cluster centers are taken to generate a set of regular grid lines, and the intersection points are calculated as the set of grid key points P_grid.

[0092] The set of key points is denoted as To improve stability, a consistency filter is applied to key points: points that deviate significantly from the main grid direction are removed (example: points whose distance from the fitted grid line is greater than 3 pixels are considered outliers).

[0093] S2-5 Estimates the homography matrix based on key points and performs perspective transformation. The table region is then subjected to perspective transformation based on key points for geometric correction, achieved by estimating the homography matrix H. Let the key points before correction be... The corresponding point on the target plane is (This can be defined by an "ideal rectangular outline" or a "regular grid point set"), and H is obtained by minimizing the reprojection error: Where π(·) is the homogeneous normalized projection. To remove outlier key points, this embodiment uses robust estimation (e.g., RANSAC) to solve H, which includes at least: interior point determination based on a threshold τ: if These are interior points (example τ = 3 pixels); Interior point ratio constraint ρ: if the number of interior points / total number of points ≥ ρ, then the model is accepted (example ρ = 0.6). Output the geometrically corrected image I_warp.

[0094] Abnormal branch: If the robust estimation cannot satisfy ρ (too few key points or too much noise), then use the minimum set of "outer border four corner points" to estimate H; if it still fails, output I_tb as a temporary normalized image and mark it "requires review - perspective correction failed".

[0095] S2-6 Edge defect repair and local structure completion: Edge defect repair and local structure completion are performed on I_warp to obtain a regularized table image I_rect. Specifically, this includes: Break endpoint localization: Extracting the line segment set L from I_warp, constructing a line segment endpoint map, and performing connectivity analysis to locate the break endpoint set E_break (e.g., if there is no continuous connection within a 5×5 neighborhood of an endpoint, it is considered a break endpoint). Line extension and reconstruction of missing line segments: For each break endpoint, using the direction of its corresponding line segment as the initial direction, a fitted straight line ℓ is obtained through straight line fitting. The missing gap is searched along the direction to generate extended line segments. Line extension constrains straightness with maximum deviation: ;

[0096] in For the line pixels or sampling points within the extended region, ε is the linearity tolerance (e.g., ε = 2 pixels). If If the extension length is reduced, the extension is abandoned and left to subsequent iterations for completion.

[0097] Structural constraints ensure mesh consistency: At least the following structural constraints are introduced during the completion process: Parallel consistency: The angle difference between adjacent horizontal line segments is no greater than δ (example δ=2°); Orthogonal consistency: The angle difference between the horizontal and vertical main directions is no greater than δ; Cell closure check: Any candidate cell area should satisfy the condition that the four sides are closed or can form a closed polygon by intersection points.

[0098] Iterative completion: When the closure check fails, perform iterative completion on the missing area until the structural constraints are met or the maximum number of iterations iter_max is reached (for example, iter_max=5); the parameters can be appropriately relaxed / tightened in each iteration (for example, relax ε=3 first and then tighten it to 2, or prioritize completing the outer border and then complete the inner mesh).

[0099] Output: A regularized table image I_rect, and a geometric reconstruction log (including H, interior point ratio, number of completion attempts, reasons for failure, etc.).

[0100] Preferably, the multimodal recognition model in S3 includes a visual encoder, a text encoder, and a cross-modal fusion module. The visual encoder outputs visual features of the table grid and layout, the text encoder outputs semantic features from the OCR text sequence, and the cross-modal fusion module performs joint alignment of visual and text features based on a cross-attention mechanism to simultaneously infer row and column structure, merged cell relationships, and preset field content. Furthermore, it outputs a field confidence score for each preset field, where the field confidence score is determined by the maximum value of the field classification probability. ;

[0101] in, For fields The corresponding model outputs logits; when merged cells or missing fields are detected, the context of adjacent cells under the merge relationship constraint is used as the condition input for field completion, and the confidence of the completion results is filtered for credibility using the field confidence.

[0102] Furthermore, S3-1 obtains text and image features, and performs OCR text acquisition: OCR is performed on I_rect to obtain a sequence of text fragments. ;in For text content, This is the bounding box on its I_rect. OCR can output line-level or word-level results; this embodiment prefers word-level results and aggregates them line by line subsequently.

[0103] Image feature extraction: Input I_rect into the visual encoder to obtain the visual feature map F_v, which contains table grid and layout information (such as lines, cell boundaries, table header style, etc.).

[0104] Text semantic feature extraction: Input the OCR text sequence into the text encoder (positional encoding is possible, such as...) After normalization, it is embedded as the layout position, and the output text feature F_t is generated.

[0105] S3-2 Multimodal Recognition Model Structure and Cross-modal Fusion: The multimodal recognition model in this embodiment includes at least a visual encoder, a text encoder, and a cross-modal fusion module. Specifically: the visual encoder outputs visual features F_v for the table grid and layout; the text encoder outputs semantic features F_t for the OCR text sequence; and the cross-modal fusion module performs joint alignment of F_v and F_t based on a cross-attention mechanism to form a fusion feature F_m, which is used to simultaneously infer row and column structure, merged cell relationships, and preset field content.

[0106] S3-3 Joint Inference: Structural information, text content information, and preset fields; Row and column structure inference: Outputting row segmentation, column segmentation, or cell grid probability map from the fused feature F_m, obtaining the set of row and column indices: , And generate a basic cell set. .

[0107] Merged cell relationship inference: Output the set of merged relationships The following factors can be used to make a comprehensive judgment: "adjacent cells belong to the same text block / visual boundaries are missing / high probability of merging".

[0108] Preset field extraction: Each cell or text segment is categorized / sequence-labeled to obtain field results (at least including accessory name, model specification, unit, and quantity). This embodiment outputs the field confidence score for each preset field, determined by the maximum field classification probability. ;in Output logits for the model corresponding to field f.

[0109] S3-4 Field completion when merging cells or missing fields: When merging cells or missing fields are detected, this embodiment uses the "context of adjacent cells under the merging relationship constraint" for completion:

[0110] Construct a context window: Centered on the target cell, take the text and structural relationships of 1-2 adjacent cells in the same row / column as the context Ctx;

[0111] Concatenate the Ctx with the target cell features and input the input to the completion head (the encoder can be shared with the main model), and output the candidate missing fields. ;

[0112] Using field confidence Perform credibility screening: If (Example) If the original value is not overwritten, only "complete suggestion + review flag" will be output; if Then write the completion result and record the source (inferred from the context).

[0113] Output: Table structure information (row and column indices, merge relationships, cell coordinate mapping) and text content information (cell text, field labels, field confidence).

[0114] Abnormal branch: If the OCR results are severely missing (e.g., the number of text fragments M is lower than the threshold or the average confidence is lower than the threshold), the weight of text features is reduced, and more reliance is placed on visual grid inference of structure; at the same time, it is marked "requires review - low OCR quality".

[0115] Furthermore, S4 reconstructs and summarizes the structured data table, S4-1 reconstructs the row and column indexes and merge relationships, and generates structured cell objects based on the Rows, Cols, Merge and cell text mappings output by S3.

[0116] cell=(row,col,rowspan,colspan,bbox,text,field_tag,conf).

[0117] For merged cells, the following rules apply: Content attribution: Text within the merged area is uniformly assigned to the main merged cell (top left corner is the main cell), and other subordinate cells are set to empty but retain references to the main cell; Field inheritance: If a field is determined to fall within a subordinate cell, it is written back to the field slot in the main cell and the "inheritance source coordinates" are recorded. This forms a structured data table `Table_struct`, which at least includes: a header area and a list of row records in the data area. Each row contains preset fields, their confidence levels, and source coordinates.

[0118] S4-2 Summary and Quantity Statistics of Similar Parts: Performs a summary of similar parts on records in the data area. Example rule:

[0119] The aggregation key Key = Norm(part name) + Norm(model specification key parameters); Norm(part name) uses dictionary normalization and synonym mapping (the same dictionary can be reused in S5); Norm(model specification key parameters) extracts key parameters such as size, material, grade from the model specification and formats them.

[0120] When counting quantities: first, unify the units (e.g., unify "pieces / items / items" to "items", or convert to the target unit according to the unit conversion table); then accumulate the quantities by the aggregation key to obtain SumQty(Key).

[0121] Abnormal branch: If the unit cannot be mapped or converted (e.g., an unknown unit appears), the row will not participate in the automatic merging statistics, the original row will be retained and marked with "unknown unit".

[0122] Preferably, in step S5, field validation and normalization processes construct a field consistency score, and the output is determined based on the consistency score and a threshold; wherein, the consistency score is obtained by weighting the results of multiple rule validations: ;

[0123] in, Indicates the first Whether the rule is approved To correspond to the weights, the rules must include at least: dictionary normalization and synonym mapping rules for part names, format template matching and key parameter extraction rules for model specifications, unit mapping and unified conversion rules, and numerical validity rules for quantities; when or confidence level of any field A review marker is generated in time, and the review marker and the corresponding field position index are carried when the editable table file and / or structured data interface data are output.

[0124] Furthermore, S5-1 constructs a field consistency score and determines whether it passes. It performs field validation and normalization on Table_struct, constructs a field consistency score, and determines whether the output passes based on the consistency score and threshold.

[0125] The consistency score is obtained by weighting the results of multiple rule validations: ;in This indicates whether the k-th rule passes or fails, with w_k representing the corresponding weight. Example settings: Part name rule weight w_name=0.25; Model / specification rule weight w_spec=0.35; Unit rule weight w_unit=0.15; Quantity rule weight w_qty=0.25; Passes. .

[0126] The rules should include at least: Part name dictionary unification and synonym mapping rules: Standardize "screw / screw nail / screw" to the dictionary standard words; if it is not in the dictionary and the edit distance / similarity is insufficient, it will be judged as unacceptable; Model specification format template matching and Key parameter extraction rules: For example, template matching such as "DNxx, Mxx, Φxx×Lxx"; if key parameters are missing or conflicting, the test will fail. Unit mapping and unified conversion rules: Map synonymous units to standard units; if conversion is possible, perform the conversion and pass the test; otherwise, fail the test. Quantity value validity rules: The quantity should be a positive number, and integers or a specified number of decimal places are allowed (e.g., ≤3 decimal places). Items exceeding the range or containing abnormal characters will be rejected.

[0127] Example of threshold determination: .when or confidence level of any field (Example) A verification mark is generated at that time.

[0128] The S5-2 review flag is carried along with the field position index. When a review is triggered, a `review_flag` is generated, carrying the corresponding field position index `pos_idx`. Index granularity example: row_id: The row number of a structured data table; col_name: Field name (part name / model / unit / quantity); cell_bbox: The field corresponds to the coordinate box of the cell; Reason: Triggering reason ( , (e.g., inability to convert units).

[0129] S5-3 outputs the standardized results. After successful verification, the standardized results are output, including at least: Editable table file: For example, XLSX includes the original table structure (including merged cells), standardized field columns, summary statistics table, and review mark column; Structured Data interface data For example, JSON / CSV / database table formats, with fields containing standardized values, confidence levels, coordinates, review flags, and reasons.

[0130] If the verification fails, the system can still output "Result with review mark", but the status will be set to NEED_REVIEW in the interface so that the business system can proceed with the manual review process.

[0131] Input / output example, example input (illustration of project configuration table content);

[0132] Header: Part Name | Model / Specification | Unit | Quantity | Remarks

[0133] Example of a data row (noise may exist after OCR):

[0134] Row 1: Screws | M8×30 ​​| Pieces | 120 | Galvanized

[0135] Line 2: Bolts | M8×30 ​​| Pieces | 80 | -

[0136] Row 3: Gasket | Φ8 | Piece | 200 | -

[0137] Example output (structured JSON fragment illustration), after standardization: Part name = Bolt ("screws / bolts" are processed according to a dictionary strategy, which can be configured to be of the same or different categories); Unit = Piece ("pieces / pieces / pieces" are processed according to a mapping table; if "pieces" cannot be directly converted to "pieces", they can be retained and verified); Quantity = 200;

[0138] Example output snippet:

[0139]

[0140] Example of abnormal output (requires review), when or :

[0141]

[0142] Abnormal branch and fault tolerance strategy: If table area detection fails, enable degenerate localization (full image or dense line segment area) and mark it as needing review; If there are insufficient key points / too many outer points, use the four corner points of the outer border to estimate H; If it still fails, skip perspective correction and mark it.

[0143] Severe edge defects: Increase iter_max or first complete the outer border; if it still cannot be closed, output a conservative structure and mark it as needing review;

[0144] Low OCR quality: Reduce text feature weights and prioritize visual grid structure inference; the output carries the reason for "low OCR quality";

[0145] Units cannot be converted / dictionary not found: Do not force normalization, retain the original value and output the verification mark and field position index.

[0146] Example 2:

[0147] refer to Figures 6-10 A multimodal engineering configuration table content recognition system is disclosed. The system runs on a computer device, which includes a processor, memory, a communication interface, and program instructions stored in the memory and executable by the processor. The processor can be one or more combinations of CPU / GPU / NPU. The engineering configuration table image can be input via a camera, scanner, or file system. The system organizes processing by task, with each task encompassing image enhancement, geometric normalization, multimodal recognition, structured reconstruction, verification, and output. Intermediate results and logs are retained to support engineering traceability and manual review. The system includes:

[0148] The task management module is used to acquire at least one image of the engineering configuration table, establish a recognition task, generate a task identifier, and record task parameters and processing status.

[0149] The adaptive image enhancement and layout geometry reconstruction module is used to evaluate the image quality of the engineering configuration table image to obtain quality features, and select an enhancement strategy to enhance the image based on the quality features; perform table region detection and extract table key points on the enhanced image; perform perspective transformation on the table region based on the table key points to perform geometric correction, and perform edge defect repair and local structure completion on the geometrically corrected image to output a regular table image.

[0150] The multimodal content recognition module is used to acquire image features and text features of a regular table image, and input them into a multimodal recognition model to jointly infer the table's row and column structure, merged cell relationships, and preset field content; wherein the preset fields include at least accessory name, model specifications, unit and quantity, and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells;

[0151] The structured reconstruction and summary statistics module is used to reconstruct a structured data table based on the row and column structure, merged cell relationship and preset field content, including at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories;

[0152] The field validation and normalization module is used to perform field validation and normalization on structured data tables, and generate standardized results after the validation passes.

[0153] The output interface module is used to output standardized results, which include at least editable table files and / or structured data interface data for import into business systems.

[0154] Preferably, the task management module is used to acquire at least one engineering configuration table image and establish a recognition task, generate a task identifier, and record task parameters and processing status.

[0155] Input and Task Object: Input: Project configuration table image I0 (can be a single image or in batches); Task parameters (such as output format, scene configuration, review threshold, etc.). The Task object must include at least: task_id, source_type, timestamp, output_mode, quality_profile, retry_limit, and status (task identifier, source type, timestamp, output mode, quality profile / scene parameter configuration, maximum number of retry attempts, task status). State Machine: INIT → PREPROCESSING → RECOGNIZING → REBUILDING → VALIDATING → OUTPUTTING → DONE / NEED_REVIEW / FAILED (Initialization → Preprocessing → Recognition in progress → Reconstruction in progress → Verification in progress → Output in progress → Completed / Requires review / Failed).

[0156] Regarding exceptions and retries, when subsequent modules return failure codes or "requires verification" results due to insufficient confidence, the task management module updates the status and records the reason for failure and key log pointers. For recoverable failures (such as network I / O failures or model loading failures), retries are performed according to retry_limit, and the log version number is retained for each retry.

[0157] As a preferred embodiment, the adaptive image enhancement and layout geometry reconstruction module is used to: perform image quality assessment on the engineering configuration table image to obtain quality features, and select enhancement strategies based on the quality features; perform table region detection and key point extraction on the enhanced image; perform perspective transformation based on the key points to complete geometric correction; and perform edge defect repair and local structure completion on the corrected image to output a regular table image.

[0158] 1. An image quality assessment submodule, in one embodiment, the image quality assessment includes at least the following quality items: sharpness, noise level, brightness / contrast offset, shadow / occlusion ratio, tilt angle, watermark / background interference intensity; and configures a threshold or grading rule for each quality item to adaptively select an enhancement strategy accordingly.

[0159] Clarity The example uses Laplace response variance. ;

[0160] noise Examples use smoothed residual variance or high-frequency energy ratio estimation;

[0161] Brightness / Contrast The example consists of the mean brightness, standard deviation, and histogram distribution offset.

[0162] Shadow / Occlusion Ratio The example shows the proportion of the shaded area to the total area of ​​the candidate areas in the table.

[0163] Incline angle Example based on the deviation angle between the peak of the principal direction of the Hough line and the horizontal / vertical direction;

[0164] Watermark / Background Interference Example: A comprehensive evaluation based on frequency domain regular texture energy concentration and low-contrast repeating connected component density.

[0165] Threshold set It can be loaded according to task parameters and scene configuration (e.g., different threshold / grading tables are used for scanning scenes and photo taking scenes), and supports configuring the priority of subsequent strategies according to task parameters.

[0166] 2. Enhancement Strategy Selection and Execution Submodule: Based on quality characteristics and threshold / grading results, the module adaptively selects enhancement strategies. Enhancement strategies include at least one or more combinations of: denoising, sharpening or deblurring, adaptive adjustment of brightness and contrast, shadow suppression, background suppression and / or watermark suppression, and supports configuring strategy priorities according to task parameters (e.g., "remove shadows first and then enhance contrast" or "denoise first and then sharpen").

[0167] Example strategies and parameters:

[0168] Denoising: Non-local means (search window 15-31, template window 5-9) or bilateral filtering (spatial σ2-6, grayscale σ20-80);

[0169] Sharpening / Deblurring: Unsharp Mask (radius 0.8-2.0, intensity 0.3-0.8) or a lightweight deblurring network;

[0170] Contrast and brightness: CLAHE (grid 4×4-16×16, crop threshold 1.5-4.0).

[0171] Shadow suppression: Background illumination surface estimation (guided filtering / Retinex class) followed by division correction;

[0172] Watermark / Background Suppression: Frequency domain notch filtering + morphological suppression or watermark separation network to remove the watermark layer.

[0173] The module outputs the enhanced image I_enh and the enhancement log EnhLog (which records the activation policy, parameters, sequence, and time consumption) for auditing and review playback.

[0174] 3. Table region detection and filtering submodule: In one embodiment, the table region detection uses at least object detection or semantic segmentation to output the table's bounding region, and performs morphological filtering and connected component filtering on the detection results to remove non-table regions.

[0175] Object detection method: Output candidate box set B={b_i}, each b_i contains coordinates and confidence score conf_i; select the one with the largest conf as b*.

[0176] Semantic segmentation method: Output table mask M_tb, denoise it by closing / opening operation and retain the largest connected component, and obtain the bounding rectangle as b*.

[0177] Morphological screening and connected component filtering: Filter candidate regions by area, aspect ratio, line density, and text density to remove watermark blocks, headers and footers, or non-table stamp areas.

[0178] Abnormal branch: If max(conf_i) is lower than the threshold conf_min, then degenerate localization is enabled (such as replacing with the densest area of ​​the largest line segment, or using the entire map as a candidate) and "requires review - table localization is uncertain" is returned to the task management module.

[0179] 4. Table key point extraction submodule: Table key points include at least the corner points of the outer border of the table and / or grid intersections; key point extraction adopts at least one of the following two methods: (1) Line segment detection and intersection solution: perform edge detection and Hough transform on the candidate table area to obtain a set of horizontal / vertical line segments, calculate the intersections and filter the corner points of the outer border and the intersections of the inner grid; (2) Prediction method based on key point network: directly predict the corner / intersection heatmap by the key point detection network, and then obtain the key point set P through non-maximum suppression. Key point anomaly handling: when the number of key points is insufficient or the distribution is unreasonable (such as missing corner points or intersections concentrated in a local area), output anomaly markers for subsequent robust sampling and consistency verification.

[0180] 5. Perspective Transformation and Geometric Correction Submodule: Perspective transformation is based on at least four corner points or multi-point fitting to obtain geometric correction parameters (homography matrix H), and robust sampling and consistency verification are used to improve correction stability when key points are abnormal.

[0181] Four-corner point scheme: H is directly solved using the four corner points of the outer border as the corresponding point set; Multi-point fitting scheme: H is solved by minimizing the reprojection error using the intersection of corner points and grid points as the corresponding point set; Robust sampling and consistency verification: RANSAC is used in the example, setting the reprojection threshold τ (2~5 pixels) and the inlier ratio threshold ρ (0.5~0.75), and H is refined after removing outliers; If the consistency verification fails, it reverts to the four-corner point scheme or reduces the degrees of freedom for affine approximation. Outputs the geometrically corrected image I_warp and the geometric log GeoLog (including H, inlier_ratio, reproj_error, and reason for reversion).

[0182] 6. Edge defect repair and local structure completion sub-module: Edge defect repair and local structure completion include at least: endpoint location of broken line segments, line segment extension, intersection reconstruction, and cell closure check; when the closure check fails, iterative completion or rollback to the backup reconstruction strategy is triggered.

[0183] Example Implementation: ① Break Endpoint Location: Construct the endpoint neighborhood connectivity relationship for the line segment set, and find the endpoints that cannot be connected to line segments in the same direction as the break endpoints; ② Line Segment Extension: Fit a straight line based on the direction of the line segment to which the break endpoint belongs and extend it, setting a straightness tolerance ε (1~4 pixels) to constrain the rationality of the extension; ③ Intersection Reconstruction: Recalculate the intersection points of the extended horizontal / vertical lines and update the grid intersection point set; ④ Cell Closure Check: Check whether the candidate cells satisfy the requirement of four-sided closure or can form a closed polygon; ⑤ Iterative Completion: If the closure fails, iteratively complete it according to the priority of "outer frame first, then inner grid", with a maximum number of iterations iter_max (3~8); if it still fails, fall back to the backup strategy (such as reducing the resolution to re-examine the line segment, using a segmentation mask to reconstruct the boundary, or outputting conservatively and marking it as needing review).

[0184] The module ultimately outputs a regularized table image I_rect (as a unified coordinate reference for subsequent multimodal recognition input) and a completion log RepairLog (including the number of completion attempts, reasons for failure, and review suggestions).

[0185] Preferably, the multimodal content recognition module is used to acquire the image features and text features of the regular table image, and input them into the multimodal recognition model to jointly infer the table row and column structure, the relationship of merged cells, and the content of preset fields; the preset fields include at least the accessory name, model specifications, unit and quantity; and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells.

[0186] In one embodiment, the multimodal content recognition module comprises: a visual encoding unit, a text recognition unit, and a cross-modal fusion inference unit.

[0187] The visual encoding unit is used to extract grid lines, cell boundaries and layout features from I_rect and output visual features F_v (such as feature maps or multi-scale feature pyramids). Visual encoding can use convolutional networks, Transformer visual backbones or a combination of both.

[0188] The text recognition unit performs optical character recognition (OCR) on I_rect, outputting the text content and its bounding box to form a text sequence. It can also normalize bbox_i to embed the layout position for use in blending and alignment.

[0189] The cross-modal fusion inference unit is used to align text location boxes with visual grid features (e.g., cross attention, gating fusion, or alignment matching), and jointly outputs: a row / column structure (row / column splitting or grid index); a merged cell relationship (including rowspan, colspan, etc.); preset field values: part name, model specification, unit and quantity; and outputs confidence C_f (e.g., maximum classification probability or sequence label confidence aggregation value) for the above field values.

[0190] Missing field completion is performed when there are merged cells or missing fields. It is based on at least the following: the context of adjacent cells in the same row / column; the coverage of the same merged cell (taking the text and structural relationship within the coverage as a condition); and field template constraints (such as model specification format templates, unit candidate set constraints, quantity numerical regular expression constraints).

[0191] Example of completion writing strategy: When the completion confidence C_f is higher than the threshold τ_C (0.6~0.85), the structured result is written; if it is lower than the threshold, only "completion candidate + verification mark" is output, without overwriting the original identification value.

[0192] As a preferred option, the structured reconstruction and summary statistics module is used to reconstruct a structured data table based on row and column structure, merged cell relationships, and preset field content. This includes at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories.

[0193] Reconstructing structured data tables Create a two-dimensional index: `row_id` and `col_id` generate a cell object `Cell(row (row index), col (column index), rowspan (row merge span), colspan (column merge span), bbox (bounding box / outer rectangle), text (text content), field_pack (field pack / field set), conf (confidence score))`.

[0194] Merge relationship: The content within the merged area is assigned to the main cell (such as the top left corner), while the reference pointers of the remaining cells are retained so that the merged structure can be exported;

[0195] Header and data separation: The header boundary can be identified based on the font style / merging pattern / keywords (such as "part name / model / unit / quantity") of the header row.

[0196] For summarizing and counting similar parts, define the aggregation key Key=Norm(name)+Norm(spec_params); Norm(name) is unified based on dictionary synonyms; Norm(spec_params) extracts key parameters (such as diameter, length, grade, material) from the model specifications and standardizes them;

[0197] Unit handling: Prioritize unit mapping and conversion; if conversion is not possible, group and statistically analyze the units and mark them as requiring review.

[0198] Output: A structured data table Table_struct (containing the reconstruction results of the original table and a summary statistics table).

[0199] As a preferred option, the field validation and normalization module is used to perform field validation and normalization processing on the structured data table, and generate standardized results after the validation passes.

[0200] 1. Dictionary and rule base driven validation standardization: In one embodiment, field validation and standardization are based on at least the dictionary and rule base to process the following: synonyms of accessory names; model specification format template matching and key parameter extraction; unit mapping and conversion; quantity validity checks (numerical range, decimal places, abnormal characters, etc.).

[0201] 2. Consistency determination and review flag generation: When the confidence level of any field is lower than the threshold τ_C, or any verification rule fails (e.g., template mismatch, unit incompatibility, illegal quantity), a review flag ReviewFlag is generated, and a review item list ReviewItems is formed.

[0202] The review items should include at least the following: row_id: the data row; col_name: the field name (part name / model specification / unit / quantity); cell_bbox: the cell position box corresponding to the field (coordinates of the regularization chart); reason: the triggering reason (such as conf_low (low confidence), template_mismatch (template mismatch), unit_mapping_failed (unit mapping failure), qty_illegal (invalid quantity)).

[0203] To support writing back to the original image, this embodiment retains the coordinate mapping (such as homography matrix or transformation chain) during the geometric reconstruction stage, so that the cell_bbox can be mapped from the I_rect coordinate system back to the I0 coordinate system, which is convenient for front-end highlight positioning.

[0204] Preferably, the output interface module is used to output standardized results, which include at least editable tabular files and / or structured data interface data for import into business systems.

[0205] 1. Output types: Editable tabular files: such as XLSX, retaining row and column structure and merged cells, with additional standardized field columns, summary statistics sheets, and review information columns; Structured interface data: such as JSON / CSV / database table records, with fields containing standardized values, confidence levels, location indexes, and review flags.

[0206] 2. Carrying a review marker and row and column position index: In one embodiment, when the output interface module outputs editable table files and / or structured data interface data, it carries a review marker and its corresponding row and column position index (e.g., row_id (row identifier), col_name (column name / field name), cell_bbox (cell bounding box), reason (reason / trigger reason)) to support the automatic routing (pass / required review) and manual verification location of the business system.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing the content of an engineering configuration table based on multimodality, characterized in that, The specific steps include the following: S1. Obtain at least one engineering configuration table image and establish a recognition task for the engineering configuration table image; S2. Perform adaptive image enhancement and layout geometric reconstruction on the project configuration table image to obtain a regular table image. The adaptive image enhancement and layout geometric reconstruction include at least: performing image quality assessment on the project configuration table image to obtain quality features; determining an enhancement strategy based on the quality features and performing image enhancement on the project configuration table image; performing table region detection and extracting table key points on the enhanced image; performing perspective transformation on the table region based on the table key points to perform geometric correction; and performing edge defect repair and local structure completion on the geometrically corrected image. S3. Perform multimodal content recognition on the regularized table image to output the table structure information and text content information of the engineering configuration table. The multimodal content recognition includes: acquiring the image features and text features of the regularized table image, and inputting them into a multimodal recognition model to jointly infer the row and column structure, the relationship between merged cells, and the content of preset fields; wherein, the preset fields include at least the accessory name, model specification, unit and quantity, and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells. S4. Reconstruct a structured data table based on the table structure information and text content information, including at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories; S5. Perform field validation and normalization on the structured data table, and output the normalization result after the validation passes. The normalization result includes at least an editable table file and / or structured data interface data for import into the business system.

2. The method for recognizing the content of a multimodal engineering configuration table according to claim 1, characterized in that, The image quality assessment in S2 quantifies the quality of the engineering configuration table image into a quality feature vector, including at least sharpness, noise, brightness / contrast, shadow occlusion, tilt, and watermark interference indicators; wherein the sharpness indicator is represented by the Laplacian operator response variance. The shadow occlusion index is expressed as the percentage of shadow area: ; in, The image grayscale matrix, For the Laplace operator, The area of ​​the shaded region. The area of ​​the table region; and, based on the quality feature vector and the preset threshold set. Adaptive selection of enhancement strategies, including at least: when Deblurring / sharpening is performed when the noise level is below a certain threshold. Denoising is performed during operation; when brightness / contrast deviates from the threshold range, illumination equalization and adaptive contrast stretching are performed. Shadow compensation is applied when the tilt index exceeds [a certain threshold]. Time-enhanced geometric correction priority, when watermark interference index exceeds Watermark suppression and background suppression are performed simultaneously.

3. The method for recognizing the content of a multimodal engineering configuration table according to claim 1, characterized in that, The table region detection in S2 outputs the detection confidence of the table candidate region, and selects the candidate region with the highest confidence as the table region; the table key points include at least the corner points of the table outer border and / or grid intersections; When performing perspective transformation based on the key points of the table, the homography matrix is ​​estimated. Achieve geometric correction, and the This is obtained by minimizing the reprojection error: ; in, Key points before calibration For the corresponding point on the target plane, For homogeneous normalized projection; and, the The solution employs robust estimation to remove outlier keypoints, and the robust estimation includes at least threshold-based methods. Interior point determination and interior point scaling constraints .

4. The method for recognizing the content of a multimodal engineering configuration table according to claim 1, characterized in that, The edge defect repair and local structure completion in S2 include: performing connectivity analysis on the geometrically corrected set of line segments to locate the broken endpoints, and extending the broken endpoints to reconstruct the missing line segments; wherein, the line extension adopts straight line fitting and constrains the straightness of the lines with maximum deviation: ; in, To fit a straight line, To extend the line pixels or sampling points within the region, To ensure linearity tolerance, structural constraints are introduced during the completion process to guarantee mesh consistency. These structural constraints include at least the following: the angle difference between adjacent horizontal line segments is no greater than [value missing]. Parallel consistency, the difference between the horizontal and vertical principal directions is no greater than The orthogonality consistency and cell closure check are performed; when the closure check fails, the missing area is iteratively filled until the structural constraints are met or the maximum number of iterations is reached.

5. The method for recognizing the content of a multimodal engineering configuration table according to claim 1, characterized in that, The multimodal recognition model in S3 includes a visual encoder, a text encoder, and a cross-modal fusion module. The visual encoder outputs visual features of the table grid and layout, the text encoder outputs semantic features from the OCR text sequence, and the cross-modal fusion module performs joint alignment of visual and text features based on a cross-attention mechanism to simultaneously infer row and column structure, merged cell relationships, and preset field content. Furthermore, it outputs a field confidence score for each preset field, where the field confidence score is determined by the maximum value of the field's classification probability. ; in, For fields The corresponding model outputs logits; when merged cells or missing fields are detected, the context of adjacent cells under the merge relationship constraint is used as the condition input for field completion, and the confidence of the completion results is filtered for credibility using the field confidence.

6. The method for recognizing the content of a multimodal engineering configuration table according to claim 1, characterized in that, The field validation and normalization process in S5 constructs a field consistency score, and the output is determined based on the consistency score and a threshold; wherein, the consistency score is obtained by weighting the results of multiple rule validations: ; in, Indicates the first Whether the rule is approved To correspond to the weights, the rules must include at least: dictionary normalization and synonym mapping rules for part names, format template matching and key parameter extraction rules for model specifications, unit mapping and unified conversion rules, and numerical validity rules for quantities; when or confidence level of any field A review marker is generated in time, and the review marker and the corresponding field position index are carried when the editable table file and / or structured data interface data are output.

7. A multimodal engineering configuration table content recognition system, characterized in that, include: The task management module is used to acquire at least one image of the engineering configuration table, establish a recognition task, generate a task identifier, and record task parameters and processing status. The adaptive image enhancement and layout geometry reconstruction module is used to perform image quality assessment on the engineering configuration table image to obtain quality features, and select an enhancement strategy to enhance the image based on the quality features. Perform table region detection and extract table key points from the enhanced image; Based on the table's key points, a perspective transformation is performed on the table area for geometric correction. Then, edge defect repair and local structure completion are performed on the geometrically corrected image to output a regular table image. The multimodal content recognition module is used to acquire image features and text features of a regular table image, and input them into a multimodal recognition model to jointly infer the table's row and column structure, merged cell relationships, and preset field content; The preset fields mentioned above include at least the accessory name, model specifications, unit and quantity, and when there are merged cells or missing fields, the missing fields are filled in based on the context of adjacent cells; The structured reconstruction and summary statistics module is used to reconstruct a structured data table based on the row and column structure, merged cell relationship and preset field content, including at least reconstructing row and column indexes and merging relationships, and performing summary and quantity statistics on similar accessories; The field validation and normalization module is used to perform field validation and normalization on structured data tables, and generate standardized results after the validation passes. The output interface module is used to output standardized results, which include at least editable table files and / or structured data interface data for import into business systems.

8. The multimodal engineering configuration table content recognition system according to claim 7, characterized in that, The image quality assessment in the adaptive image enhancement and layout geometry reconstruction module includes at least the following verifiable quality items: sharpness, noise level, brightness / contrast shift, shadow / occlusion ratio, tilt angle, and watermark / background interference intensity; and configures a corresponding threshold or grading rule for each quality item to adaptively select an enhancement strategy accordingly; the enhancement strategy includes at least one or more combinations of denoising, sharpening or deblurring, adaptive adjustment of brightness and contrast, shadow suppression, background suppression, and / or watermark suppression, and supports configuring the strategy priority according to task parameters.

9. The multimodal engineering configuration table content recognition system according to claim 7 or 8, characterized in that, The table region detection employs at least object detection or semantic segmentation to output the table's bounding region, and performs morphological screening and connected component filtering on the detection results to eliminate non-table regions; the table key points include at least the corner points of the table's outer border and / or grid intersections, and key point extraction employs at least line segment detection and intersection solving or prediction based on key point networks; the perspective transformation is based at least on four corner points or multi-point fitting to obtain geometric correction parameters, and robust sampling and consistency verification are used to improve correction stability when key points are abnormal; the edge defect repair and local structure completion include at least broken line segment endpoint localization, line segment extension, intersection reconstruction, and cell closure check, and when the closure check fails, iterative completion or regression to a backup reconstruction strategy is triggered.

10. The multimodal engineering configuration table content recognition system according to claim 7 or 8, characterized in that, The multimodal content recognition module includes a visual encoding unit, a text recognition unit, and a cross-modal fusion inference unit. Visual encoding units are used to extract grid lines, cell boundaries, and layout features; The text recognition unit is used to perform text recognition on a regular table image and output the text content and its location box; The cross-modal fusion inference unit is used to align text location boxes and visual grid features, jointly output row and column structure, merged cell relationship, and field values ​​of accessory name, model specification, unit and quantity, and output confidence scores for the field values; Among them, missing field completion is based on inference and completion at least on adjacent cells in the same row / column, the coverage of the same merged cells, and field template constraints. The field validation and normalization module performs processing on at least the dictionary and rule base for synonym unification of accessory names, matching of model specifications and format templates, unit mapping and conversion, and quantity legality checks, and generates a review mark when the confidence level is lower than the threshold or the validation fails; the output interface module carries the review mark and its corresponding row and column position index when outputting editable table files and / or structured data interface data.

Images