A method and system for chart extraction and chart-text pairing oriented to engineering specification documents

Abstract

The application relates to the technical field of layout analysis and artificial intelligence interaction, and provides a chart extraction and image-text pairing method and system for engineering specification documents.The method comprises the following steps: eliminating illumination gradients by background decoupling and image binarization on the obtained document image, extracting a text line skeleton to generate a feature point set, deriving and constructing a 3D curved surface model of paper according to the feature point set, and obtaining a flattened image through reverse mapping; treating the flattened image by using a horizontal and vertical separation RLSA algorithm, extracting the external contour of all macro connected domains and generating the corresponding minimum circumscribed rectangle, then uniformly drawing on a pure black background to generate a physical boundary connected domain; based on the document image, a SSA-YOLO target detection network is used to obtain a semantic prediction box; if the area intersection ratio between the physical boundary connected domain and the semantic prediction box is greater than a set coincidence threshold, the category of the semantic prediction box is assigned to the corresponding physical boundary connected domain, and the positioning and classification of a chart main body are completed.

Description

Technical Field

[0001] This invention relates to the field of layout analysis and artificial intelligence interaction technology, and in particular to a method and system for extracting charts and matching text and images in engineering specification documents. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the deepening of industrial informatization and industry digital transformation, various industrial fields have accumulated a massive amount of unstructured engineering specification documents. These documents exist in the form of standards, guidelines, and construction specifications, containing not only a large amount of text but also rich examples, diagrams, and tables, carrying engineering structures and data logic that are difficult to express in plain text. High-precision layout analysis and chart extraction of these documents are crucial prerequisites for constructing industrial knowledge graphs, indexing engineering atlases, and supporting the construction of large-scale language model knowledge bases for vertical domains. However, engineering specification documents possess highly unique and complex characteristics in terms of physical carriers and layout paradigms. How to accurately identify, locate, and extract chart elements and establish their association with names has become a pressing technical challenge that needs to be addressed.

[0004] While existing deep learning-based layout analysis methods have made some progress, they still have significant shortcomings when dealing with complex engineering specification documents. Specifically: First, physical degradation introduced by photography or scanning, such as uneven lighting, perspective distortion, 3D curling, and creases and shadows, disrupts the original neat layout of the document, leading to false detections when using deep learning detection algorithms. Second, the highly dense layout of charts and explanatory text makes it difficult for existing detection algorithms to distinguish the physical boundaries between adjacent charts, easily resulting in merged selections and misidentifying surrounding dense text blocks or background noise as chart elements. Third, because the chart content and its name in a specification document may have a large spatial span, most algorithms can only output isolated target bounding boxes, lacking the ability to analyze the spatial correspondence of layout elements, making it difficult to accurately match "figures with figure names" and "tables with table names" in complex layouts. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a method and system for extracting charts and matching text and images in engineering specification documents. This invention provides systematic improvements from image preprocessing, dense layout segmentation, small target feature enhancement to text-image logical association, significantly enhancing the overall accuracy and robustness of complex engineering specification document layout parsing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for extracting charts and matching text and images in engineering specification documents.

[0007] A method for extracting charts and matching text with images in engineering specification documents, including: The acquired document image is decoupled from the background and binarized to obtain a binarized image. The text line skeleton of the binarized image is extracted to generate a feature point set. The four virtual reference corner points of the paper plane are calculated. The projection relationship of the virtual reference corner points is solved to infer the external parameters when the camera is shooting. A 3D curved surface model of the paper is constructed. The flattened image is obtained through reverse mapping. The RLSA algorithm, which separates horizontal and vertical lines, is used to process the flattened image, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles. The minimum bounding rectangles are then drawn uniformly on a pure black background to generate physical boundary connected components that have eliminated background noise interference. Based on the document image, the SSA-YOLO object detection network is used to obtain semantic prediction boxes; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. If the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than the set overlap threshold, the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thus completing the location and classification of the main body of the chart.

[0008] Furthermore, the acquired document image is subjected to background decoupling and image binarization processing to obtain a binarized image; the method includes: The acquired document image is converted into a grayscale image, and morphological dilation and median filtering are performed on the grayscale image to obtain the document background image; The difference between the document background image and the document image is calculated and inverted to generate a differential image with balanced illumination. The difference image is linearly normalized and stretched to grayscale. Then, the maximum inter-class variance method is used to traverse the grayscale levels of the entire image to find the grayscale boundary that maximizes the inter-class difference as the optimal segmentation threshold. This completes the binarization segmentation of the image and outputs the binarized image.

[0009] Further, the method for extracting the text line skeleton from the binarized image to generate a feature point set includes: The text line skeleton of the binarized image is extracted, and the coordinates of the text line skeleton are sampled at equal intervals to generate a feature point set. The text line skeleton refers to the continuous text centerline trajectory formed by concatenating discrete characters after extracting the minimum bounding rectangle, centroid, and local tilt features of the characters in the binarized image. This trajectory is used to characterize local paper curling and page tilt features. The continuous text centerline trajectory is calculated by minimizing the edge connectivity cost function. The edge connectivity cost function is expressed by the following formula:

[0010] in, This represents the straight-line distance between the centers of any two nodes A and B; The penalty weight for angle difference, Indicates the deviation of the tilt angle; The weights are for non-overlapping penalties in the vertical direction; This indicates the non-overlapping ratio of the two-character rectangles in the vertical direction.

[0011] Furthermore, the method involves calculating four virtual reference corner points on the paper plane, solving the projection relationship of these virtual reference corner points to infer the external parameters at the time of camera capture, constructing a 3D curved surface model of the paper, and obtaining a flattened image through reverse mapping; the method includes: The feature point set is analyzed to extract the horizontal and vertical main directions of the paper. Combined with the estimated physical aspect ratio of the paper, four virtual reference corner points of the paper plane are calculated. The projection relationship of these virtual reference corner points is solved, and the rotation and translation vectors during camera shooting are derived to initialize the camera's external parameters. Based on the physical law that paper curls only in the horizontal direction while remaining straight in the vertical direction, the pixel coordinates in the horizontal direction of the image are linearly mapped to a normalized horizontal coordinate range. A cubic polynomial is used to construct a 3D surface model of the paper. The 3D surface model is expressed by the following formula:

[0012] in, and Unknown shape parameters that determine the curling shape of paper; Based on the model parameters of the 3D surface model, predicted surface points are generated in three-dimensional space. Combining the camera intrinsic parameter matrix, rotation vector, and translation vector, the geometric distance deviation function between the predicted surface points and the true feature points is calculated. This geometric distance deviation function is expressed by the following formula:

[0013] in, Indicates the predicted surface point, Represents real feature points, , Represents the rotation and translation vectors during camera shooting. Represents the camera intrinsic parameter matrix; Through fine-tuning and To minimize the geometric distance deviation function, the theoretical projection of the predicted surface points coincides with the real feature points to the greatest extent. Based on the obtained physical dimensions of the page, a flat two-dimensional target mesh is constructed. For each pixel coordinate on the mesh, the 3D surface model and camera parameters are substituted to calculate the exact corresponding position coordinates of the pixel in the document image, thus completing pixel remapping and obtaining a flattened image.

[0014] Furthermore, the method employs the RLSA algorithm with horizontal and vertical separation to process the flattened image; the method includes: A morphological closing operation is performed on the flattened image in the horizontal direction to connect the discrete characters in the horizontal direction into a single-line text connected component; Based on the horizontal smooth connectivity results, the vertical merging threshold is dynamically set, and morphological closing operation is performed again in the vertical direction to vertically merge the discrete chart physical blocks into a complete macroscopic connected domain. At the same time, line spacing is used to block the erroneous adhesion of single-line text in the vertical direction.

[0015] Furthermore, the SPD convolutional module consists of an SPD periodic transformation layer and a standard convolutional layer with a stride of 1 connected in series. The input feature map is sliced ​​at equal intervals in the spatial dimension by the SPD periodic transformation layer, splitting it into several sub-feature map sequences. Then, the several sub-feature map sequences are concatenated along the channel dimension to generate an intermediate feature map with high channel density. Finally, the channel information is fused and the feature dimension is adjusted by the standard convolutional layer with a stride of 1. The SimAM attention module utilizes the local statistics of the feature map itself, including the mean and variance, to define an energy function for each feature point to evaluate its linear separability. It calculates the reciprocal of the energy matrix as the three-dimensional attention weight and scales it to a standard range through an activation function. Finally, it performs element-wise adaptive recalibration on the original input features.

[0016] Further, the method of assigning the category of the semantic prediction box to the corresponding physical boundary connected component includes: If the target category is used as an example, the graph name is searched downwards based on the bottom edge of the connected domain. If the target category is used as an example table, then the table name is searched upwards based on the top edge of the connected component. Meanwhile, based on the average row height of the text connected components around the chart, an adaptive distance threshold is dynamically calculated to determine a reasonable search range, and only text connected components within this distance range are included in the chart name candidate set.

[0017] A second aspect of the present invention provides a system for extracting charts and matching text and images in engineering specification documents.

[0018] A system for extracting diagrams and matching text with images in engineering specification documents includes: The image preprocessing and 3D flattening module is configured to: obtain a binarized image by decoupling the acquired document image from the background and binarizing the image; extract the text line skeleton of the binarized image to generate a feature point set; calculate the four virtual reference corner points of the paper plane; solve the projection relationship of the virtual reference corner points to infer the external parameters when the camera was shooting; construct a 3D curved surface model of the paper; and obtain the flattened image through reverse mapping. The RLSA algorithm module is configured to: process the flattened image using the horizontal and vertical separation RLSA algorithm, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles, draw the minimum bounding rectangles uniformly on a pure black background, and generate physical boundary connected components that have eliminated background noise interference. The SSA-YOLO object detection network module is configured to: obtain semantic prediction boxes based on document images using the SSA-YOLO object detection network; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. The chart element extraction and semantic association module is configured to: if the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than a set overlap threshold, then the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thereby completing the positioning and classification of the chart subject.

[0019] A third aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the diagram extraction and text-image matching method for engineering specification documents as described in the first aspect above.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute the steps of the diagram extraction and text-image matching method for engineering specification documents as described in the first aspect above.

[0021] Compared with the prior art, the beneficial effects of the present invention are: This invention first eliminates lighting distortion through 3D surface reconstruction to restore a regular layout. Then, it uses the RLSA algorithm to generate macroscopic physical boundary connected components to accurately distinguish the boundaries of densely arranged charts. Simultaneously, in the SSA-YOLO backbone network, the downsampled SPD convolution module preserves fine-grained features and combines a parameter-free SimAM attention module to enhance important signals. Finally, the IoA (Intersection over Union) of physical connected components and semantic prediction boxes is used to jointly determine the category assignment and region attribution. Thus, it not only suppresses false detections caused by physical degradation from the source and solves the problems of merging box selection and background misjudgment under dense layout, but also effectively establishes the logical association between the chart subject and its corresponding name by relying on the large-span spatial coverage provided by the physical boundary, realizing the accurate positioning, classification and pairing of chart elements in complex engineering specification documents. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart illustrating a method for extracting charts and matching text and images in engineering specification documents, as shown in an embodiment of the present invention. Figure 2 This is a flowchart of another embodiment of the method for extracting charts and matching text and images in engineering specification documents, as shown in the embodiments of the present invention; Figure 3 This is a flowchart illustrating the extraction of chart elements and the matching of text and images, as shown in an embodiment of the present invention. Figure 4 This is a structural diagram of a diagram extraction and image-text matching system for engineering specification documents, as shown in an embodiment of the present invention. Figure 5 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Figure 1 This is a flowchart illustrating a method for extracting charts and matching text and images in engineering specification documents, as shown in an embodiment of the present invention; see reference. Figure 1 The method includes: The acquired document image is decoupled from the background and binarized to obtain a binarized image. The text line skeleton of the binarized image is extracted to generate a feature point set. The four virtual reference corner points of the paper plane are calculated. The projection relationship of the virtual reference corner points is solved to infer the external parameters when the camera is shooting. A 3D curved surface model of the paper is constructed. The flattened image is obtained through reverse mapping. The RLSA algorithm, which separates horizontal and vertical lines, is used to process the flattened image, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles. The minimum bounding rectangles are then drawn uniformly on a pure black background to generate physical boundary connected components that have eliminated background noise interference. Based on the document image, the SSA-YOLO object detection network is used to obtain semantic prediction boxes; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. If the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than the set overlap threshold, the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thus completing the location and classification of the main body of the chart.

[0028] This invention provides systematic improvements from image preprocessing, dense layout segmentation, small target feature enhancement to text-image logical association, significantly enhancing the overall accuracy and robustness of complex engineering specification document layout parsing.

[0029] Figure 2 This is a flowchart illustrating another embodiment of the method for extracting charts and matching text and images from engineering specification documents, as shown in this invention; see also... Figure 2The method for extracting charts and matching text and images in engineering specification documents mainly includes four steps: image preprocessing and 3D flattening, physical boundary extraction of layout elements based on the Run Length Smoothing Algorithm (RLSA), layout analysis model based on SSA-YOLO, and extraction and semantic association of chart elements. Each step is described in detail below.

[0030] Step 1: Image preprocessing and 3D flattening.

[0031] To address common physical distortions and uneven lighting issues in original scanned or photographed documents, this method eliminates lighting gradients through background decoupling and image binarization, extracts text line skeletons to generate feature point sets, derives and constructs a 3D curved surface model of the paper, and corrects the curved page to a flat view through reverse mapping. The text line skeleton refers to the continuous text centerline trajectory formed by concatenating discrete characters after extracting the minimum bounding rectangle, centroid, and local tilt features of the characters in the binarized document image; it is used to characterize local paper curling and page tilt features. The specific workflow is as follows: First, the original document image is converted to a grayscale image. Then, a structuring element of a preset size is applied to dilate the grayscale image to eliminate foreground text features. Finally, a median blur algorithm is used for filtering to obtain the estimated document background image B. (1) The difference between the document background image B and the original image I is calculated and inverted to generate a differential image D with balanced illumination, thereby increasing the contrast between the foreground text and the background and suppressing low-brightness shadow interference. (2) The difference image is linearly normalized and stretched to enhance the sharpness of text edges. Finally, the maximum inter-class variance method is used to traverse the gray levels of the entire image to find the gray level boundary that maximizes the inter-class difference as the optimal segmentation threshold, thus completing the binarization segmentation of the image and outputting the binarized image J. (3) Extract the minimum bounding rectangle of each character from the binarized image to determine the character centroid, and calculate the centroid coordinates of each character using image spatial moments. Then, the second-order central moment covariance matrix of the character is constructed and singular value decomposition is performed. The feature vector corresponding to the largest singular value is extracted as the tangent direction vector of the local text block. This is used to characterize the degree of local tilt caused by the curling of paper in a single character; each isolated character is considered as a centroid node, and the straight-line distance between the centers of any two nodes A and B is comprehensively evaluated. Alignment along the tilt direction and tilt angle deviation Based on this, the edge connectivity cost function is constructed: (4) in, The penalty weight for angle difference, The weights are for non-overlapping penalties in the vertical direction; This indicates the non-overlapping ratio of the two-character rectangles in the vertical direction.

[0032] By finding the optimal path with the minimum total cost among reasonable connections, discrete characters are assembled into continuous text lines. The tilt features of the paper are extracted, and coordinate sampling is performed at equal intervals along these skeleton trajectories to generate a feature point set S for subsequent 3D surface modeling. The feature point set is analyzed to extract the horizontal and vertical principal directions of the paper. Combined with the estimated physical aspect ratio of the paper, four virtual reference corner points of the paper plane are calculated. The projection relationship of these virtual reference corner points is solved, and the rotation vector during camera capture is deduced. With translation vector To complete the initialization of camera external parameters; based on the physical law that paper curls only in the horizontal direction while remaining straight in the vertical direction, the pixel coordinates in the horizontal direction of the image are linearly mapped to the normalized horizontal coordinate range, and a cubic polynomial is used to construct the 3D surface model of the paper: (5) In the formula, and To determine the unknown shape parameters that determine the curling pattern of paper, predicted surface points are generated in three-dimensional space based on the model parameters. Combined with camera intrinsic parameter matrix The rotation vector and translation vector Calculate and predict surface points Compared with real two-dimensional feature points Geometric distance deviation function between: (6) Through fine-tuning and To minimize the deviation function, so that the predicted surface points Theoretical projection and real feature points To achieve maximum overlap and restore the true three-dimensional curled state of the paper; based on the obtained physical dimensions of the page, a flat two-dimensional target grid is constructed. For each pixel coordinate on the grid, the 3D surface model and camera parameters are substituted to calculate the exact corresponding position coordinates of the pixel in the original curved image, complete the pixel remapping, and output the flattened and corrected image.

[0033] This invention constructs a 3D curved surface model of paper and flattens the image by background decoupling, binarization, and virtual reference corner point projection back calculation. This effectively eliminates the physical degradation effects such as uneven lighting, perspective distortion, three-dimensional curling, and crease shadows introduced by taking pictures or scanning, providing a regular input image for subsequent detection and reducing the false detection rate caused by page distortion from the source.

[0034] Step 2: Extraction of physical boundaries of layout elements based on RLSA.

[0035] The horizontal and vertical separation RLSA algorithm is used to process the flattened image. First, morphological closing operations are performed on the flattened and corrected image in the horizontal direction, connecting discrete characters across the gaps between characters in the same row into a single-line text connected component. Then, based on the horizontal smooth connectivity result, a vertical merging threshold is dynamically set. Specifically, the vertical spacing between adjacent lines of text in the page is extracted, and the median of the vertical spacing is calculated. Determine the vertical merging threshold according to the preset ratio coefficient. ,Right now =

[0036] Among them, the proportionality coefficient The value range is 0.1 to 0.4. By setting a vertical merging threshold, morphological closing operations are performed again in the vertical direction to vertically merge the discrete physical blocks of the chart into complete macroscopic connected components. At the same time, line spacing is used to prevent erroneous vertical connections of single-line text, avoiding vertical misconnections in dense layouts. After completing the above RLSA connectivity processing, the outer contours of all macroscopic connected components are extracted and corresponding minimum bounding rectangles are generated. The minimum bounding rectangles are uniformly drawn on a pure black background to generate clean physical boundary connected components of the layout, free from background noise interference.

[0037] This invention employs the RLSA algorithm with horizontal and vertical separation to extract macroscopic connected components and generate the minimum bounding rectangle. After filtering out background noise, the physical boundary connected components are obtained. This process can precisely distinguish the boundaries between adjacent charts, avoiding the merging and selection of multiple independent charts in dense layouts, while effectively eliminating interference from surrounding dense text blocks and background noise.

[0038] Step 3: Layout analysis model based on SSA-YOLO.

[0039] Next, the document image is input into the improved SSA-YOLO (SPD and SimAM-YOLO) object detection network. This network preserves local minute details by introducing an SPD lossless downsampling convolutional module into the backbone network and cascading a SimAM attention mechanism to enhance the network's response weights to key text regions, reduce the influence of background elements, and accurately output high-confidence semantic prediction boxes for charts and their names. The specific work is as follows: In the backbone of the object detection network, all the original hard downsampling strided convolutional nodes are replaced with Space-to-Depth (SPD) convolutional modules to avoid feature loss caused by strided convolution when processing dense layouts. The SPD convolutional module consists of an SPD periodic transformation layer and a standard convolutional layer with a stride of 1 connected in series. The input feature map is sliced ​​at equal intervals in the spatial dimension by the periodic transformation layer, splitting it into several sub-feature map sequences. Then, the sub-feature map sequences are concatenated along the channel dimension to generate a high-channel-density intermediate feature map. Then, the channel information is fused and the feature dimension is adjusted by the standard convolutional layer with a stride of 1, achieving downsampling without discarding any original feature pixels.

[0040] After the deep feature extraction of the backbone network is completed, the SimAM attention module is connected in series on the output path to guide the network to focus on key chart information and suppress complex background noise. The SimAM attention module directly uses the local statistics of the feature map itself, including the mean and variance, to define an energy function for each feature point to evaluate its linear separability. It uses the reciprocal of the energy matrix obtained by fast calculation as the three-dimensional attention weight and scales it to the standard range through the activation function. Finally, it performs element-wise adaptive recalibration on the original input features.

[0041] The configured SSA-YOLO network was iteratively trained on a layout analysis dataset pre-labeled with illustrations, tables, figure titles, and table titles. The converged model was then used to perform forward inference for object detection on the flattened image, outputting predicted bounding boxes with high-confidence category semantics and location boundary information.

[0042] This invention replaces the downsampling strided convolution with an SPD convolution module in the backbone network of the SSA-YOLO model, preserving fine-grained information in the feature map and solving the problem of information loss for small targets caused by traditional downsampling. Simultaneously, it integrates a SimAM attention module, which enhances important features and suppresses background interference without increasing the number of parameters. The synergistic effect of these two modules significantly improves the recall and localization accuracy for small, feature-indistinct chart elements in engineering specification documents.

[0043] Step 4: Extracting chart elements and matching text and images.

[0044] Figure 3 This is a flowchart illustrating the extraction of chart elements and text-image pairing in an embodiment of the present invention; see reference. Figure 3 For the same input image, the area intersection-union ratio (IoA) between the predicted bounding box output by the SSA-YOLO model and the physical boundary connected component extracted by the RLSA connectivity model is calculated. When the calculated overlap is greater than the IoA threshold (recommended value 0.8), the category label of the semantic predicted bounding box is directly assigned to the corresponding physical boundary connected component to complete the accurate localization and classification of the main body of the chart.

[0045] Specifically, the area intersection-union ratio (IoA) between the predicted bounding box output by the SSA-YOLO model and the connected domain of the physical boundary extracted by RLSA connectivity is calculated using the following formula:

[0046] in, Represents the YOLO prediction box; The IoA represents the candidate macroscopic connected components extracted by RLSA, and the IoA represents the spatial overlap between the YOLO prediction box and the candidate macroscopic connected components. The overlap threshold is the minimum overlap standard used to determine whether the semantic prediction box and the physical boundary connected components correspond to the same chart target. Its value ranges from 0 to 1, and is preferably 0.8.

[0047] The name retrieval direction is determined based on the category labels assigned to the connected components of the physical boundary. If the target category is an example image, the search for image names is performed downwards from the bottom edge of the connected component; if the target category is an example table, the search for table names is performed upwards from the top edge of the connected component. Simultaneously, an adaptive distance threshold is dynamically calculated based on the average row height of the text connected components surrounding the image to determine a reasonable search range. Only text connected components within this distance range are included in the image name candidate set. The adaptive distance threshold refers to the maximum vertical distance allowed between a candidate text connected component and the boundary of the image body in the name retrieval direction. When the vertical distance between a candidate text connected component and the boundary of the image body is less than or equal to the adaptive distance threshold, the text connected component is included in the image name candidate set; when the vertical distance is greater than the adaptive distance threshold, the text connected component is excluded.

[0048] This invention assigns semantic categories to corresponding physical regions by jointly determining the IoA (Intersection over Union) of physical boundary connected components and SSA-YOLO semantic prediction boxes. This not only achieves accurate positioning and classification of the main body of the chart, but also provides accurate regional attribution for cross-spatial map / table name associations through the complete coverage capability of macroscopic physical connected components. It effectively overcomes the shortcomings of existing algorithms that only output isolated bounding boxes and lack spatial correspondence analysis capabilities, and achieves high-precision pairing of "map and map name" and "table and table name".

[0049] To further ensure the completeness of chart extraction, three supplementary rules were designed.

[0050] ① Centered Formula Anti-interference Rule: Check whether there are multiple independent text connected components in a specific row within the candidate set of chart names. If there are other independent connected components in the same row, the area is determined to be a formula or text and is removed to ensure that the finally extracted chart name connected component occupies a single row and is centered.

[0051] ② Merging rules for peripheral edge details: Taking the initially located physical boundary of the chart as the core, calculate the horizontal and vertical spacing between the surrounding unclassified small connected regions and the boundary; when the horizontal and vertical spacing are both less than the preset merging threshold, determine that the small connected region is an auxiliary element such as the chart's outer coordinate axis, label, size number or symbol, and merge the outer boundaries of the two to complete the chart's integrity.

[0052] ③ Transitional area information merging rules: Further investigate the spatial range of the angle between the physical boundary of the main body of the chart and the successfully matched connected domain. If there are unclassified residual connected domains within the spatial range, they are determined to be legends or supplementary explanatory texts, and are forcibly merged into the overall chart structure. Finally, the semantically paired and informationally complete chart structured parsing data is output.

[0053] The above combination Figure 1 The method for extracting charts and matching text and images in engineering specification documents provided in this embodiment of the invention has been described in detail. Next, the system for extracting charts and matching text and images in engineering specification documents provided in this embodiment of the invention will be described in conjunction with the accompanying drawings.

[0054] Figure 4 This is a schematic diagram illustrating the structure of a diagram extraction and image-text matching system for engineering specification documents, as shown in an embodiment of the present invention. Figure 4 The system described in this invention includes: The image preprocessing and 3D flattening module is configured to: obtain a binarized image by decoupling the acquired document image from the background and binarizing the image; extract the text line skeleton of the binarized image to generate a feature point set; calculate the four virtual reference corner points of the paper plane; solve the projection relationship of the virtual reference corner points to infer the external parameters when the camera was shooting; construct a 3D curved surface model of the paper; and obtain the flattened image through reverse mapping. The RLSA algorithm module is configured to: process the flattened image using the horizontal and vertical separation RLSA algorithm, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles, draw the minimum bounding rectangles uniformly on a pure black background, and generate physical boundary connected components that have eliminated background noise interference. The SSA-YOLO object detection network module is configured to: obtain semantic prediction boxes based on document images using the SSA-YOLO object detection network; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. The chart element extraction and semantic association module is configured to: if the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than a set overlap threshold, then the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thereby completing the positioning and classification of the chart subject.

[0055] In some embodiments, the image preprocessing and 3D flattening module is specifically configured to: convert the acquired document image into a grayscale image, and perform morphological dilation and median filtering operations on the grayscale image to obtain a document background image; calculate the difference between the document background image and the document image and perform color inversion processing to generate a differential image with balanced illumination; perform linear normalized grayscale stretching on the differential image, and then use the maximum inter-class variance method to traverse the grayscale levels of the entire image to find the grayscale boundary that maximizes the inter-class difference as the optimal segmentation threshold, complete the binarization segmentation of the image, and output the binarized image.

[0056] In some embodiments, the image preprocessing and 3D flattening module is further configured to: extract the text line skeleton of the binarized image, perform equidistant coordinate sampling on the text line skeleton, and generate a feature point set; the text line skeleton refers to the continuous text centerline trajectory formed by concatenating discrete characters after extracting the minimum bounding rectangle, centroid, and local tilt features of the characters in the binarized image, which is used to characterize the local paper curling and page tilt features; the continuous text centerline trajectory is obtained by minimizing the edge connection cost function; the edge connection cost function is expressed by the following formula:

[0057] in, This represents the straight-line distance between the centers of any two nodes A and B; The penalty weight for angle difference, Indicates the deviation of the tilt angle; The weights are for non-overlapping penalties in the vertical direction; This indicates the non-overlapping ratio of the two-character rectangles in the vertical direction.

[0058] In some embodiments, the image preprocessing and 3D flattening module is further configured to: analyze the feature point set, extract the horizontal and vertical main directions of the paper, and calculate the four virtual reference corner points of the paper plane based on the estimated physical aspect ratio of the paper; solve the projection relationship of the virtual reference corner points; and deduce the rotation and translation vectors during camera shooting to complete the camera's external parameter initialization; based on the physical law that paper only curls in the horizontal direction and remains straight in the vertical direction, linearly map the pixel coordinates in the horizontal direction of the image to the normalized horizontal coordinate range, and construct a 3D surface model of the paper using a cubic polynomial; the 3D surface model is expressed by the following formula:

[0059] in, and Unknown shape parameters that determine the curling shape of paper; Based on the model parameters of the 3D surface model, predicted surface points are generated in three-dimensional space. Combining the camera intrinsic parameter matrix, rotation vector, and translation vector, the geometric distance deviation function between the predicted surface points and the true feature points is calculated. This geometric distance deviation function is expressed by the following formula:

[0060] in, Indicates the predicted surface point, Represents real feature points, , Represents the rotation and translation vectors during camera shooting. Represents the camera intrinsic parameter matrix; Through fine-tuning and To minimize the geometric distance deviation function, the theoretical projection of the predicted surface points coincides with the real feature points to the greatest extent. Based on the obtained physical dimensions of the page, a flat two-dimensional target mesh is constructed. For each pixel coordinate on the mesh, the 3D surface model and camera parameters are substituted to calculate the exact corresponding position coordinates of the pixel in the document image, thus completing pixel remapping and obtaining a flattened image.

[0061] In some embodiments, the RLSA algorithm module is specifically configured to: perform morphological closing operations on the flattened image in the horizontal direction to connect discrete characters in the horizontal direction into a single-line text connected component; based on the horizontal smooth connectivity result, dynamically set a vertical merging threshold, and perform morphological closing operations again in the vertical direction to vertically merge the discrete chart physical blocks into a complete macroscopic connected component, while using line spacing to block erroneous adhesion of single-line text in the vertical direction.

[0062] In some embodiments, the SPD convolutional module consists of an SPD periodic transformation layer and a standard convolutional layer with a stride of 1 connected in series. The input feature map is sliced ​​at equal intervals in the spatial dimension by the SPD periodic transformation layer, splitting it into several sub-feature map sequences. Then, the several sub-feature map sequences are concatenated along the channel dimension to generate an intermediate feature map with high channel density. Finally, the channel information is fused and the feature dimension is adjusted by the standard convolutional layer with a stride of 1.

[0063] In some embodiments, the SimAM attention module utilizes the local statistics of the feature map itself, including the mean and variance, to define an energy function for each feature point to evaluate its linear separability, calculates the reciprocal of the energy matrix as the three-dimensional attention weight, and scales it to a standard range through an activation function, and finally performs element-wise adaptive recalibration on the original input features.

[0064] In some embodiments, assigning the category of the semantic prediction box to the corresponding physical boundary connected component includes: if the target category is an example graph, searching for the graph name downwards based on the bottom edge of the connected component; if the target category is an example table, searching for the table name upwards based on the top edge of the connected component; and simultaneously, dynamically calculating an adaptive distance threshold based on the average row height of the text connected components surrounding the graph to determine a reasonable search range, and only extracting the text connected components within this distance range into the graph name candidate set.

[0065] According to embodiments of the present invention, the diagram extraction and image-text matching system for engineering specification documents can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the diagram extraction and image-text matching system for engineering specification documents are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0066] See Figure 5The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to achieve the corresponding steps in the embodiment of the method for extracting diagrams and matching text and images in engineering specification documents.

[0067] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device.

[0068] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0069] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above embodiment of the method for extracting charts and matching text and images for engineering specification documents.

[0070] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0071] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting charts and matching text and images in engineering specification documents, characterized in that, include: The acquired document image is decoupled from the background and binarized to obtain a binarized image. The text line skeleton of the binarized image is extracted to generate a feature point set. The four virtual reference corner points of the paper plane are calculated. The projection relationship of the virtual reference corner points is solved to infer the external parameters when the camera is shooting. A 3D curved surface model of the paper is constructed. The flattened image is obtained through reverse mapping. The RLSA algorithm, which separates horizontal and vertical lines, is used to process the flattened image, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles. The minimum bounding rectangles are then drawn uniformly on a pure black background to generate physical boundary connected components that have eliminated background noise interference. Based on the document image, the SSA-YOLO object detection network is used to obtain semantic prediction boxes; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. If the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than the set overlap threshold, the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thus completing the location and classification of the main body of the chart.

2. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The method involves obtaining a binary image by decoupling the acquired document image from the background and performing image binarization; the method includes: The acquired document image is converted into a grayscale image, and morphological dilation and median filtering are performed on the grayscale image to obtain the document background image; The difference between the document background image and the document image is calculated and inverted to generate a differential image with balanced illumination. The difference image is linearly normalized and stretched to grayscale. Then, the maximum inter-class variance method is used to traverse the grayscale levels of the entire image to find the grayscale boundary that maximizes the inter-class difference as the optimal segmentation threshold. This completes the binarization segmentation of the image and outputs the binarized image.

3. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The method for extracting the text line skeleton from the binarized image to generate a feature point set includes: The text line skeleton of the binarized image is extracted, and the coordinates of the text line skeleton are sampled at equal intervals to generate a feature point set. The text line skeleton refers to the continuous text centerline trajectory formed by concatenating discrete characters after extracting the minimum bounding rectangle, centroid, and local tilt features of the characters in the binarized image. This trajectory is used to characterize local paper curling and page tilt features. The continuous text centerline trajectory is calculated by minimizing the edge connectivity cost function. The edge connectivity cost function is expressed by the following formula: in, This represents the straight-line distance between the centers of any two nodes A and B; The penalty weight for angle difference, Indicates the deviation of the tilt angle; The weights are for non-overlapping penalties in the vertical direction; This indicates the non-overlapping ratio of the two-character rectangles in the vertical direction.

4. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The method involves calculating four virtual reference corner points on the paper plane, solving the projection relationship of these virtual reference corner points to infer the external parameters at the time of camera capture, constructing a 3D curved surface model of the paper, and obtaining a flattened image through reverse mapping. The method includes: The feature point set is analyzed to extract the horizontal and vertical main directions of the paper. Combined with the estimated physical aspect ratio of the paper, four virtual reference corner points of the paper plane are calculated. The projection relationship of these virtual reference corner points is solved, and the rotation and translation vectors during camera shooting are derived to initialize the camera's external parameters. Based on the physical law that paper curls only in the horizontal direction while remaining straight in the vertical direction, the pixel coordinates in the horizontal direction of the image are linearly mapped to a normalized horizontal coordinate range. A cubic polynomial is used to construct a 3D surface model of the paper. The 3D surface model is expressed by the following formula: in, and Unknown shape parameters that determine the curling shape of paper; Based on the model parameters of the 3D surface model, predicted surface points are generated in three-dimensional space. Combining the camera intrinsic parameter matrix, rotation vector, and translation vector, the geometric distance deviation function between the predicted surface points and the true feature points is calculated. This geometric distance deviation function is expressed by the following formula: in, Indicates the predicted surface point, Represents real feature points, , Represents the rotation and translation vectors during camera shooting. Represents the camera intrinsic parameter matrix; Through fine-tuning and To minimize the geometric distance deviation function, the theoretical projection of the predicted surface points coincides with the real feature points to the greatest extent. Based on the obtained physical dimensions of the page, a flat two-dimensional target mesh is constructed. For each pixel coordinate on the mesh, the 3D surface model and camera parameters are substituted to calculate the exact corresponding position coordinates of the pixel in the document image, thus completing pixel remapping and obtaining a flattened image.

5. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The method employs the RLSA algorithm, which separates horizontal and vertical axes, to process the flattened image; the method includes: A morphological closing operation is performed on the flattened image in the horizontal direction to connect the discrete characters in the horizontal direction into a single-line text connected component; Based on the horizontal smooth connectivity results, the vertical merging threshold is dynamically set, and morphological closing operation is performed again in the vertical direction to vertically merge the discrete chart physical blocks into a complete macroscopic connected domain. At the same time, line spacing is used to block the erroneous adhesion of single-line text in the vertical direction.

6. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The SPD convolutional module consists of an SPD periodic transformation layer and a standard convolutional layer with a stride of 1 connected in series. The input feature map is sliced ​​at equal intervals in the spatial dimension by the SPD periodic transformation layer, splitting it into several sub-feature map sequences. Then, the several sub-feature map sequences are concatenated along the channel dimension to generate an intermediate feature map with high channel density. Finally, the channel information is fused and the feature dimension is adjusted by the standard convolutional layer with a stride of 1. The SimAM attention module utilizes the local statistics of the feature map itself, including the mean and variance, to define an energy function for each feature point to evaluate its linear separability. It calculates the reciprocal of the energy matrix as the three-dimensional attention weight and scales it to a standard range through an activation function. Finally, it performs element-wise adaptive recalibration on the original input features.

7. The method for extracting charts and matching text and images in engineering specification documents according to claim 1, characterized in that, The method of assigning the category of the semantic prediction box to the corresponding physical boundary connected component includes: If the target category is used as an example, the graph name is searched downwards based on the bottom edge of the connected domain. If the target category is used as an example table, then the table name is searched upwards based on the top edge of the connected component. Meanwhile, based on the average row height of the text connected components around the chart, an adaptive distance threshold is dynamically calculated to determine a reasonable search range, and only text connected components within this distance range are included in the chart name candidate set.

8. A system for extracting charts and matching text with images in engineering specification documents, characterized in that, include: The image preprocessing and 3D flattening module is configured to: obtain a binarized image by decoupling the acquired document image from the background and binarizing the image; extract the text line skeleton of the binarized image to generate a feature point set; calculate the four virtual reference corner points of the paper plane; solve the projection relationship of the virtual reference corner points to infer the external parameters when the camera was shooting; construct a 3D curved surface model of the paper; and obtain the flattened image through reverse mapping. The RLSA algorithm module is configured to: process the flattened image using the horizontal and vertical separation RLSA algorithm, extract the outer contours of all macroscopic connected components and generate the corresponding minimum bounding rectangles, draw the minimum bounding rectangles uniformly on a pure black background, and generate physical boundary connected components that have eliminated background noise interference. The SSA-YOLO object detection network module is configured to: obtain semantic prediction boxes based on document images using the SSA-YOLO object detection network; in the backbone network of the SSA-YOLO object detection network, all downsampling strided convolutional nodes are replaced with SPD convolutional modules, and SimAM attention modules are connected in series on the output path. The chart element extraction and semantic association module is configured to: if the area intersection-union ratio between the physical boundary connected component and the semantic prediction box is greater than a set overlap threshold, then the category of the semantic prediction box is assigned to the corresponding physical boundary connected component, thereby completing the positioning and classification of the chart subject.

9. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the diagram extraction and image-text matching method for engineering specification documents as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the diagram extraction and text-image matching method for engineering specification documents as described in any one of claims 1-7.

Images