A document table structure recognition method for enhanced retrieval of large water conservancy models

Abstract

This application provides a document table structure recognition method for enhancing large-scale water conservancy model retrieval, relating to the field of water conservancy document table processing and analysis. It solves the technical problem in large-scale model retrieval enhancement techniques where existing document recognition technologies struggle to accurately extract the large amounts of complex, multi-page, unstructured table information contained within water conservancy documents. The method includes: acquiring a document X containing water conservancy information, splitting it into several single pages, and dividing each page into several column regions and several row regions; extracting the visual information vector and text information vector of each row region and generating a joint feature vector; obtaining the cross-row feature matrix between row regions using UNet; and based on the cross-row feature matrix, obtaining the label of each row region and merging the row regions. This application is used in the process of processing water conservancy document tables.

Description

Technical Field

[0001] This application relates to the field of water conservancy document table processing and analysis, and in particular to a document table structure recognition method for enhancing large-scale water conservancy model retrieval. Background Technology

[0002] In the field of water conservancy engineering, technical documents, monitoring reports, and design specifications commonly contain a large amount of high-density and highly complex tabular data, such as precipitation statistics tables, flow tables, and design specification tables. These documents are mostly presented in formatted formats, and the tabular data they contain often spans multiple pages.

[0003] Accurately extracting tabular data from these documents and transforming it into structured documents is a crucial and challenging task for information integration and utilization in the field of water conservancy engineering. However, existing general-purpose models often fail to address the highly specialized content of the water conservancy field due to a lack of in-depth analysis and thorough understanding of the industry's context. In practical applications, this often results in irrelevant answers or even semantic illusions, significantly hindering the accurate extraction of tabular data from water conservancy documents. Summary of the Invention

[0004] This application provides a document table structure recognition method for enhancing large-scale water conservancy model retrieval, which solves the technical problem that existing document recognition technologies are insufficient to accurately extract the large amount of complex, multi-page unstructured table information contained in water conservancy documents.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a document table structure recognition method for enhancing large-scale water conservancy model retrieval is provided, including:

[0007] Document X containing water resources information was split into several single pages in its original order. ;in, Indicates the page number. , Indicates the total number of pages. ;

[0008] Each page is divided into several basic grids; each basic grid includes several column areas. and several rows of areas Where j represents the column number and k represents the row number;

[0009] Extract each row region using a fully convolutional neural network. Visual information vector And extracting text information vectors through LayoutLM Visual information vector Visual information and text information vectors representing each row region. This represents the text information for each line region; where, and These are the feature dimensions for visual features and text features, respectively;

[0010] Visual information vectors and text information vector Concatenate along the feature dimension to generate a joint feature vector. ;

[0011] Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix ;in, The number of feature channels, This represents the total number of rows in the current column range.

[0012] Based on the cross row feature matrix Get each row region Tags;

[0013] Based on each row region Tags, merged line areas .

[0014] Based on the above technical solution, in the document table structure recognition method for enhanced retrieval of large-scale water conservancy models provided in this application, the visual information vector of each row region is first extracted using a fully convolutional neural network, and the text information vector is extracted using LayoutLM. Then, the two are concatenated along the feature dimension to generate a joint feature vector. This processing method fully integrates information from two different modalities, text and vision, enabling the text content in the table to be closely combined with corresponding visual features such as border lines. This effectively avoids the problem of losing cross-modal semantic associations, thus enabling better extraction of cross-page, multi-format table data in documents, especially suitable for complex tables commonly found in water conservancy information documents. By establishing internal structure labels to merge regions, flexible judgment and merging can be performed based on the actual characteristics and structure of the table data, greatly improving the effect of merging regions and better adapting to the complex and varied format characteristics of water conservancy professional table data, ensuring accurate extraction of table data under different layout conditions.

[0015] In conjunction with the first aspect above, in one possible implementation, the plurality of column regions The steps to obtain it include:

[0016] Split pages using a splitting model For multiple basic grids ;in, Indicates the total number of rows. Indicates the total number of columns. , ;

[0017] A merging model is used to merge at least one basic grid into a single column region, resulting in multiple column regions; wherein, for several basic grids that need to be merged into a single column region... After merging, the column area is obtained. For basic meshes that do not need to be merged , column area .

[0018] In conjunction with the first aspect above, in one possible implementation, the row region The steps to obtain it include:

[0019] Row regions are divided according to physical object category. Classified as a character line area Vector row region and blank line areas;

[0020] contiguous character line regions Merge into character line area This yields a set of character line regions. , will be a continuous vector row region Merge into vector row regions This yields the set of vector row regions. .

[0021] In conjunction with the first aspect above, in one possible implementation, the physical object includes, but is not limited to, at least one of text characters, numerical symbols, table lines, graphic elements, and blank areas.

[0022] In conjunction with the first aspect above, in one possible implementation, the extraction of each row region is performed using a fully convolutional neural network. Visual information vector ,include:

[0023] The document page image is encoded using FCN to generate a feature map aligned with the spatial dimensions of the original image, dividing each line region... The feature embeddings are pooled to obtain the visual information vector. This is used to represent the visual information of each row area.

[0024] In conjunction with the first aspect above, in one possible implementation, the row-region-based approach... joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix ,include:

[0025] The UNet model encoder obtains the joint feature vector of each row region. Four downsampling operations are performed using convolution-max pooling to obtain four levels of intermediate feature maps;

[0026] The UNet model decoder performs four upsampling operations on the highest-level feature map output by the encoder through deconvolution to obtain a four-level feature map.

[0027] The intermediate feature maps of each level of the encoder are concatenated and fused with the corresponding level feature maps obtained after deconvolution using a skip-connection method along the channel dimension to obtain the cross-row feature matrix. .

[0028] In conjunction with the first aspect above, in one possible implementation, the feature matrix based on the cross rows... Get each row region The tags include:

[0029] Predefine physical object categories and corresponding state labels, and construct several category-state combinations;

[0030] Will Each row is mapped to a 3N-dimensional vector through a fully connected layer, and then the probability distribution is calculated using a softmax function. The category-state combination corresponding to the highest probability value is selected as each row region. The label; where N is the number of predefined physical object categories.

[0031] In conjunction with the first aspect mentioned above, in one possible implementation, based on each row region Tags, merged line areas ,include:

[0032] The header row serves as the starting point for a new table; if consecutive data rows... [k,k1]> If the rows continue across pages, they will be merged into the same table; [k,k1]> If the rows are not part of the table, they will be merged into the table containing the last row of the previous page; the merged data will be a structured table; non-table content rows will not be merged. and These are the preset merging thresholds for data rows and cross-page rows, respectively; k is the data row currently being processed, and k1 is the data row being compared with the current row k.

[0033] In conjunction with the first aspect above, in one possible implementation, the column region The horizontal projections of all physical objects do not intersect;

[0034] The row area The vertical projections of all physical objects do not intersect.

[0035] Secondly, a document table structure recognition device for enhancing large-scale water conservancy model retrieval is provided, comprising: a communication unit and a processing unit; the communication unit is used to acquire a document X containing water conservancy information and split the document into several single pages in its original order. The processing unit is used to divide each page into several basic grids; each basic grid includes several columns of regions. and several rows of areas Extract each row region Visual information vector and text information vector ; Visual information vector and text information vector Concatenate along the feature dimension to generate a joint feature vector. Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix Based on the cross row feature matrix Get each row region Tags; based on each row region Tags, merged line areas .

[0036] Thirdly, this application provides a document table structure recognition device for enhancing large-scale hydraulic model retrieval, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is used to execute the instructions to implement the method described in the first aspect and any possible implementation thereof. This document table structure recognition device for enhancing large-scale hydraulic model retrieval can be an electronic device or a chip within an electronic device.

[0037] Fourthly, this application provides a document table structure recognition system for enhancing retrieval of large-scale water conservancy models, comprising: a segmentation module, an information extraction module, and a merging module; wherein, the segmentation module is used to obtain document X containing water conservancy information and split the document into several single pages in the original order. Divide each page into several basic grids; each basic grid includes several column areas. and several rows of areas The information extraction module is used to extract each row region using a fully convolutional neural network. Visual information vector And extracting text information vectors through LayoutLM ; Visual information vector and text information vector Concatenate along the feature dimension to generate a joint feature vector. The merge module is used for row-based regions. joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix Based on the cross row feature matrix Get each row region Tags; based on each row region Tags, merged line areas .

[0038] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a document table structure recognition device for enhanced retrieval of large-scale water conservancy models, cause the document table structure recognition device for enhanced retrieval of large-scale water conservancy models to perform the method described in the first aspect and any possible implementation thereof.

[0039] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a document table structure recognition device for enhanced retrieval of large-scale water conservancy models, causes the document table structure recognition device for enhanced retrieval of large-scale water conservancy models to perform the method described in the first aspect and any possible implementation thereof.

[0040] This application provides a document table structure recognition method for enhanced retrieval of large-scale water conservancy models. It can accurately identify and extract table data content from water conservancy documents. First, the document is split into single pages in its original order, and then further divided into column and row regions. This refined hierarchical processing method can accurately locate the smallest information unit in the document, laying the foundation for accurate data extraction. Second, the fusion of multimodal information is a major highlight. Visual and textual information vectors of the row regions are extracted using a fully convolutional neural network and LayoutLM, respectively, and then concatenated into a joint feature vector. This fully considers the relationship between text and visual elements such as borders in the table, avoiding cross-modal semantic loss and effectively handling cross-page and multi-format tables. Finally, in region merging, the cross-row feature matrix is ​​obtained based on the joint feature vector, and then the row region labels are determined and merged. This method overcomes the limitations of relying on heuristic rules and can flexibly judge according to the actual table structure, making it more adaptable to complex formatted tables in water conservancy fields and greatly improving the accuracy and completeness of data extraction.

[0041] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0042] Figure 1 A system architecture diagram of a document table structure recognition system for enhancing retrieval of large water conservancy models provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating a document table structure recognition method for enhancing retrieval of large-scale water conservancy models provided in this application embodiment;

[0044] Figure 3 This is a schematic diagram of the RPN model structure based on multi-dilation factor convolution provided in the embodiments of this application;

[0045] Figure 4 This application provides a schematic diagram of the encoding structure for document page image encoding.

[0046] Figure 5 This is a schematic diagram of the structure of an identification device provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the hardware structure of an identification device provided in an embodiment of this application. Detailed Implementation

[0048] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0049] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0050] The document table structure recognition method for enhanced retrieval of large-scale water conservancy models provided in this application can be applied to, for example... Figure 1 The document table structure recognition system shown is an enhancement for large-scale water conservancy model retrieval. Figure 1 As shown, the system includes: a segmentation module, an information extraction module, and a merging module.

[0051] The segmentation module is used to split the document X containing water conservancy information into several single pages in its original order. ;in, Indicates the page number. , Indicates the total number of pages. ;

[0052] Each page is divided into several basic grids; each basic grid includes several column areas. and several rows of areas Where j represents the column number and k represents the row number.

[0053] The information extraction module is used to extract each row region. Visual information vector and text information vector ; Visual information vector and text information vector Concatenate along the feature dimension to generate a joint feature vector. ;

[0054] The merge module is used for merging based on row regions. joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix Based on the cross row feature matrix Get each row region Tags; based on each row region Tags, merged line areas .

[0055] The information extraction module is communicatively connected to both the segmentation module and the merging module.

[0056] To address the technical challenge of accurately extracting large amounts of complex, multi-page unstructured table information from water conservancy documents using existing document recognition techniques in large-scale model retrieval enhancement technologies, this application provides a document table structure recognition method for water conservancy large-scale model retrieval enhancement. This method includes: acquiring a document X containing water conservancy information, and splitting the document into several single pages in its original order. Divide each page into several basic grids; each basic grid includes several column areas. and several rows of areas Extract each row region using a fully convolutional neural network. Visual information vector And extracting text information vectors through LayoutLM Visual information vector Visual information and text information vectors representing each row region. Representing the text information of each line region; vectorizing the visual information. and text information vector Concatenate along the feature dimension to generate a joint feature vector. Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix Based on the cross row feature matrix Get each row region Tags; based on each row region Tags, merged line areas Based on this, the document is first split into single pages in its original order to ensure the information order remains intact; then, each page is divided into column and row regions. This hierarchical and refined structured splitting method is like building a clear coordinate system for the document, accurately locating every smallest information unit within it, providing a solid and reliable foundation for accurate extraction of table data and effectively avoiding information confusion and omissions. In the information extraction stage, a fully convolutional neural network is used to extract the visual information vectors of the row regions, and LayoutLM is used to obtain the text information vectors. Then, the two are concatenated along the feature dimension to form a joint feature vector. This fusion method fully considers the close relationship between text content and visual elements such as borders and lines in the water conservancy table, breaking the limitations of traditional methods that process text or visual information separately, effectively avoiding cross-modal semantic loss, and greatly enhancing the ability to understand the semantics of complex water conservancy tables. Based on the joint feature vector, the cross-row feature matrix is ​​obtained through UNet, thereby determining the label of each row region and completing the merging. This process does not rely on fixed heuristic rules, but rather performs intelligent analysis and judgment based on the actual characteristics and structure of the table. It can flexibly handle complex layout situations commonly found in water conservancy professional tables, such as tables spanning multiple pages and irregular row and column distributions, significantly improving the accuracy and adaptability of regional merging and ensuring the high-quality completion of water conservancy information table data extraction.

[0057] like Figure 2 As shown in the embodiment of this application, a document table structure recognition method for enhancing large-scale water conservancy model retrieval includes:

[0058] S201. Document X containing water conservancy information is split into several single pages in its original order. .

[0059] in, Indicates the page number. , Indicates the total number of pages. .

[0060] For example, a Python library can be used to preserve the original bookmarks and comments, inherit the original document's XMP metadata, and maintain a strict page order index.

[0061] For example, a water resources information document X, such as a hydrological monitoring report or engineering drawings, may contain multiple pages, cross-page tables, and mixed text and graphics content. It is a collection of single pages split in the original order. Page numbers and layout information are retained.

[0062] S202. Divide each page into several basic grids; each basic grid includes several column areas. and several rows of areas Where j represents the column number and k represents the row number.

[0063] S203. Extract each row region using a fully convolutional neural network. Visual information vector And extract each row region using LayoutLM Text information vector ;

[0064] Among them, visual information vector Visual information and text information vectors representing each row region. This represents the text information for each line area; and These are the feature dimensions for visual features and textual features, respectively.

[0065] S204: Transform visual information vectors and text information vector Concatenate along the feature dimension to generate a joint feature vector. .

[0066] S205: Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix ;

[0067] in, The number of feature channels, This represents the total number of rows in the current column range.

[0068] S206: Based on the cross row feature matrix Get each row region The tag.

[0069] S207: Based on each row region Tags, merged line areas .

[0070] Based on the above technical solutions, this application provides a document table structure recognition method for enhancing large-scale water conservancy model retrieval. In terms of information location, it constructs a rigorous and logically clear hierarchical splitting system. First, it splits the document into single pages according to its original order, completely preserving the original structure and information order, laying an orderly foundation for subsequent processing. Then, each page is divided into several column and row areas. This progressively refined splitting method, from the whole to the part, is like drawing a precise map for the document, ensuring that each table element is accurately located, effectively avoiding extraction errors caused by inaccurate information location, and providing a reliable guarantee for accurate table data extraction. At the semantic understanding level, it possesses powerful multimodal information fusion capabilities. It deeply mines the visual information of the row areas through a fully convolutional neural network, while simultaneously using LayoutLM to accurately capture textual information, and concatenates the two along the feature dimension to generate a joint feature vector. This fusion method achieves deep interaction and complementarity between visual and textual information, fully considering the close relationship between text content and visual elements such as table borders and lines in water conservancy tables. When dealing with complex layouts, it demonstrates intelligent feature analysis and merging capabilities. Based on joint feature vectors, this invention uses UNet to obtain cross-row feature matrices, then assigns labels to each row region and completes the merging. Unlike traditional methods that rely on fixed heuristic rules, it can dynamically analyze and judge based on the actual characteristics and structure of the table. This invention can flexibly adapt to complex layout situations common in water conservancy tables, such as multi-page tables, irregular row and column distributions, and mixed formats, accurately identifying table structure and semantic relationships, achieving reasonable merging of row regions, ensuring the integrity and accuracy of data extraction, and demonstrating excellent performance in processing various complex water conservancy table data.

[0071] In one possible implementation of this application embodiment, the above-mentioned S202 can be specifically implemented by the following S301 and S302, which are described in detail below:

[0072] S301, Split pages using a splitting model For multiple basic grids ;

[0073] in, Indicates the total number of rows. Indicates the total number of columns. , ;

[0074] It should be noted that large areas that span multiple columns horizontally or vertically are not considered at this point; they are still divided into basic grids.

[0075] For example, the splitting model takes an image of arbitrary dimensions H×W as input and produces two one-dimensional output signals: and The output signals r and c represent the probability that each row (column) pixel belongs to the logical table row (column) separator region.

[0076] The model is split into three sub-networks: a shared fully convolutional network (SFCN), a row projection network (RPN), and a column projection network (CPN). SFCN computes local image features that are used by both RPN and CPN. RPN and CPN further process these local features to predict row and column separators (r and c, respectively).

[0077] SFCN consists of 3 convolutional layers, each with a kernel size of 7. 7. The last layer performs a dilated convolution with a dilation factor of 2. Each layer produces 18 feature maps and uses the ReLU activation function.

[0078] like Figure 3 As shown, the RPN consists of 5 linked blocks, each block's input being fed (in parallel) into 3 convolutional layers, each with a dilation factor of 2 / 3 / 4 to produce 6 feature maps. The outputs of each dilated convolution are concatenated to obtain 18 feature maps. Next, the RPN performs 1... 2. Max pooling (CPN executes 2) After max pooling (1), through 1 1. Perform convolution operation, then projective pooling, to calculate row features.

[0079] CPN and RPN have the same structure; the projection and pooling operations are simply replaced for column pixels.

[0080] S302. Use the merging model to merge at least one basic grid into a column region to obtain multiple column regions;

[0081] Among them, for several basic grids that need to be merged into a single column region After merging, the column area is obtained. For basic meshes that do not need to be merged , column area .

[0082] For example, the merge model uses the input image and the output of the split model to predict which grid elements need to be merged to restore cells that span multiple rows or columns.

[0083] Its input consists of a table image, row / column probabilities (r, c), inferred row / column regions, and a connection to the predicted grid structure. The predicted probabilities r and c are transformed into a two-dimensional image through stacking.

[0084] The architecture of the merged model is similar to that of the split model, with a set of 4 shared convolutional layers (without dilation), of which 2 Average pooling is performed after layers 2 and 4. Afterward, the model has four branches, each predicting an M... An N matrix contains the probability that cells are merged in a specific direction, namely, up, down, left, or right.

[0085] Based on the above technical solution, a splitting model is used to divide the page into basic grids, a processing method that offers high flexibility. This splitting model can adaptively determine the total number of rows and columns according to the actual content of the page, accurately dividing the page into appropriate basic grids. This provides a unified and standardized foundation for subsequent processing, greatly improving the method's adaptability to different types of water resources documents. After obtaining the basic grids, not all grids need to remain independent; some semantically or visually related grids should be merged into a single column area to more accurately restore the actual structure of the table. The entire process from page splitting to column area merging is coherent and complete, providing strong support for the accurate extraction of water resources information table data.

[0086] In one possible implementation of this application embodiment, the above-mentioned S202 further includes the following steps S401 and S402, which are described in detail below:

[0087] S401: Divide the row area according to the type of physical object. If it includes character objects such as text, numbers, and symbols, the row area is called the character row area. If it does not include character objects but includes vector objects such as lines, polygons, and color blocks, then this row area is called a vector row area. If none of them are included, then the row area is classified as a blank row area.

[0088] The physical objects include, but are not limited to, at least one of the following: text characters, numerical symbols, table lines, graphic elements, and blank areas.

[0089] S402: Transfer consecutive character line regions Merge into character line area This yields a set of character line regions. , will be a continuous vector row region Merge into vector row regions This yields the set of vector row regions. .

[0090] Based on the above technical solution, by clearly distinguishing between character line regions, vector line regions, and blank line regions, the essential characteristics of each line region can be accurately identified, avoiding confusion between different types of objects during processing. This greatly improves the efficiency and accuracy of data processing, laying a solid foundation for efficient extraction of water conservancy information. Furthermore, adjacent character line regions or vector line regions often have semantic or visual relevance and may collectively express a complete information unit. Merging consecutive character line regions into a single character line region and consecutive vector line regions into a single vector line region integrates scattered information into a more meaningful whole. The resulting sets of character line regions and vector line regions have a clearer and more rational structure, and their information organization is more efficient.

[0091] In one possible implementation of this application embodiment, the above-mentioned S203 can be specifically described as follows:

[0092] The document page image is encoded using FCN to generate a feature map aligned with the spatial dimensions of the original image, dividing each line region... The feature embeddings are pooled to obtain the visual information vector. This is used to represent the visual information of each row area.

[0093] For example, such as Figure 4 As shown, the encoder consists of 5 downsampling modules, each of which is as follows: Figure 4 As shown, it contains 2 layers and 3 floors. 3 convolutions (stride 1, padding 1) + ReLU activation + 2 Max pooling (stride 2) is used to capture high-level semantic features. Then, spatial resolution is gradually restored through 5 upsampling modules (transposed convolution + skip connections), and finally, a feature map with the same size as the input image is output.

[0094] Based on the above technical solution, FCN is used to encode document page images. When processing water conservancy information document page images, it can comprehensively capture various visual elements within the row area, such as the font, size, and color of text, the line thickness and shape of tables, and the texture of the background. By generating feature maps aligned with the spatial dimensions of the original image, a rich and comprehensive visual feature library is constructed for each row area, laying a solid foundation for the accurate extraction of visual information from the row area. Embedding the features of each row area into a pooling operation is an efficient information extraction method. The pooling operation can reduce the dimensionality of a large amount of information in the row area feature map, remove redundant information, and retain the most representative features.

[0095] In one possible implementation of this application embodiment, the above-mentioned S205 can be specifically described as follows:

[0096] The UNet model encoder obtains the joint feature vector of each row region. Four downsampling operations are performed using convolution-max pooling to obtain four levels of intermediate feature maps;

[0097] The UNet model decoder performs four upsampling operations on the highest-level feature map output by the encoder through deconvolution to obtain a four-level feature map.

[0098] The intermediate feature maps of each level of the encoder are concatenated and fused with the corresponding level feature maps obtained after deconvolution using a skip-connection method along the channel dimension to obtain the cross-row feature matrix. .

[0099] In one possible implementation of this application embodiment, the above-mentioned S207 can be specifically described as follows:

[0100] Predefine physical object categories and corresponding state labels, and construct several category-state combinations;

[0101] Will Each row is mapped to a 3N-dimensional vector through a fully connected layer, and then the probability distribution is calculated using a softmax function. The category-state combination corresponding to the highest probability value is selected as each row region. The tag.

[0102] Where N is the number of predefined physical object categories, and each category corresponds to three state labels: starting line, inner line, and single line. For a document containing N physical object categories, the output dimension of the classification layer is expanded to 3N, that is, each category corresponds to three independent sub-state spaces.

[0103] For example, for each predefined physical object category, such as tables, legends, and annotation boxes, three status labels are defined: starting row, inner row, and single row. That is, for the starting row of a table object, its label is "table-starting row"; the row area other than the starting row is labeled "table-inner row"; if the table object contains only one row area, the label of that row area is "table-single row".

[0104] It should be noted that when dealing with documents containing more or fewer physical object categories, the output dimension of the classification layer will automatically expand or shrink by simply adjusting the value of N accordingly, without requiring large-scale modifications to the overall architecture of the model.

[0105] Based on the above technical solution, water conservancy information documents contain a rich variety of physical objects, such as text, table lines, and graphic symbols, each of which may be in different states within the document. This can be addressed by defining each predefined... NThe system assigns three status labels to physical object categories: starting line, inner line, and single line. This comprehensively and accurately describes the actual structure and semantic information of line regions within the document, making the classification results more closely reflect the document's reality and providing a reliable classification foundation for subsequent accurate extraction and processing of water conservancy information. In this 3N-dimensional vector, each dimension may represent feature information related to a specific physical object category and its status. This mapping fully preserves and extracts key information from the original features, enabling subsequent classification decisions to be based on richer and more representative features, thereby improving the accuracy and reliability of classification.

[0106] In one possible implementation of this application embodiment, the above-mentioned S207 can be specifically described as follows:

[0107] Based on each row region Tags, merged line areas .

[0108] The header row serves as the starting point for a new table; if consecutive data rows... [k,k1]> If the rows continue across pages, they will be merged into the same table; [k,k1]> If the rows are not part of the table, they will be merged into the table containing the last row of the previous page; the merged data will be a structured table; non-table content rows will not be merged. and These are the preset merging thresholds for data rows and cross-page rows, respectively; k is the data row currently being processed, and k1 is the data row being compared with the current row k.

[0109] The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, for example, a document table structure recognition device for enhanced retrieval of large-scale hydraulic models, includes at least one of the hardware structures and software modules corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] This application embodiment can divide a document table structure recognition device for enhanced retrieval of large-scale water conservancy models into functional units based on the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0111] When using integrated units, Figure 5 The above embodiment shows a possible structural diagram of a document table structure recognition device (referred to as recognition device 50) for enhancing the retrieval of large-scale water conservancy models. The recognition device 50 includes a processing unit 501 and a communication unit 502, and may also include a storage unit 503. Figure 5 The structural diagram shown can be used to illustrate the structure of the identification device involved in the above embodiments.

[0112] when Figure 5 The schematic diagram shown is used to illustrate the structure of the identification device involved in the above embodiments. The processing unit 501 is used to control and manage the operation of the identification device, the communication unit 502 is used for the identification device to communicate with other devices, and the storage unit 503 is used to store the program code and data of the identification device.

[0113] For example, communication unit 502, used to acquire water conservancy information document X, splits the document into several single pages in the original order. ;

[0114] Processing unit 501 is used to divide each page into several basic grids; each basic grid includes several column areas. and several rows of areas Extract each row region Visual information vector and text information vector ; Visual information vector and text information vector Concatenate along the feature dimension to generate a joint feature vector. Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix Based on the cross row feature matrix Get each row region Tags; based on each row region Tags, merged line areas .

[0115] The processing unit 501 can be a processor or a controller, and the communication unit 502 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 503 can be a memory. When the identification device 50 is a chip, the processing unit 501 can be a processor or a controller, and the communication unit 502 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 503 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0116] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the identification device 50 can be considered as the communication unit 502 of the identification device 50, and the processor with processing functions can be considered as the processing unit 501 of the identification device 50. Optionally, the device in the communication unit 502 that implements the receiving function can be considered as a communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 502 that implements the transmitting function can be considered as a transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0117] Figure 5 If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0118] Figure 5 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0119] This application also provides a hardware structure diagram of a document table structure recognition device (denoted as recognition device 60) for enhancing the retrieval of large-scale water conservancy models. See [link to diagram]. Figure 6 The identification device 60 includes a processor 601, and optionally, a memory 602 connected to the processor 601.

[0120] In the first possible implementation, see Figure 6 The identification device 60 also includes a transceiver 603. The processor 601, memory 602, and transceiver 603 are connected via a bus. The transceiver 603 is used to communicate with other devices or communication networks. Optionally, the transceiver 603 may include a transmitter and a receiver. The device in the transceiver 603 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 603 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0121] Based on the first possible implementation method Figure 6 The structural diagram shown can be used to illustrate the structure of the identification device involved in the above embodiments.

[0122] in, Figure 6 Alternatively, the system chip in the identification device can be illustrated. In this case, the actions performed by the identification device can be implemented by the system chip, and the specific actions performed can be found above, so they will not be repeated here.

[0123] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0124] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a SoC (System-on-a-Chip), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0125] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0126] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0127] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0128] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0130] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0131] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0132] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

Claims

1. A document table structure recognition method for enhanced retrieval of large-scale water conservancy models, characterized in that, include: Document X containing water resources information was split into several single pages in its original order. ;in, Indicates the page number. , Indicates the total number of pages. ; Each page is divided into several basic grids; each basic grid includes several column areas. and several rows of areas Where j represents the column number and k represents the row number; Extract each row region using a fully convolutional neural network. Visual information vector And extract each row region using LayoutLM Text information vector Visual information vector Visual information and text information vectors representing each row region. This represents the text information for each line region; where, and These are the feature dimensions for visual features and text features, respectively; Visual information vectors and text information vector Concatenate the features to generate a joint feature vector. ; Based on row region joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix ;in, The number of feature channels, This represents the total number of rows in the current column range. Based on the cross row feature matrix Get each row region Tags; Based on each row region Tags, merged line areas .

2. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The several columns of regions The steps to obtain it include: Split pages using a splitting model For multiple basic grids ;in, Indicates the total number of rows. Indicates the total number of columns. , ; A merging model is used to merge at least one basic grid into a single column region, resulting in multiple column regions; wherein, for several basic grids that need to be merged into a single column region... After merging, the column area is obtained. For basic meshes that do not need to be merged , column area .

3. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The row area The steps to obtain it include: Row regions are divided according to physical object category. Classified as a character line area Vector row region and blank line areas; contiguous character line regions Merge into character line area This yields a set of character line regions. , will be a continuous vector row region Merge into vector row regions This yields the set of vector row regions. .

4. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 3, characterized in that, The physical objects include, but are not limited to, at least one of the following: text characters, numerical symbols, table lines, graphic elements, and blank areas.

5. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The extraction of each row region using a fully convolutional neural network Visual information vector ,include: The document page image is encoded using FCN to generate a feature map aligned with the spatial dimensions of the original image, dividing each line region... The feature embeddings are pooled to obtain the visual information vector. This is used to represent the visual information of each row area.

6. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The row region-based joint eigenvectors Obtain the row region via UNet Intersecting row feature matrix ,include: The UNet model encoder obtains the joint feature vector of each row region. Four downsampling operations are performed using convolution-max pooling to obtain four levels of intermediate feature maps; The UNet model decoder performs four upsampling operations on the highest-level feature map output by the encoder through deconvolution to obtain a four-level feature map. The intermediate feature maps of each level of the encoder are concatenated and fused with the corresponding level feature maps obtained after deconvolution using a skip-connection method along the channel dimension to obtain the cross-row feature matrix. .

7. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The feature matrix based on cross rows Get each row region The tags include: Predefine physical object categories and corresponding state labels, and construct several category-state combinations; cross row feature matrix Each row is mapped to a 3N-dimensional vector through a fully connected layer, and then the probability distribution is calculated using a softmax function. The category-state combination corresponding to the highest probability value is selected as each row region. The label; where N is the number of predefined physical object categories.

8. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 1, characterized in that, The basis for each row region Tags, merged line areas ,include: The header row serves as the starting point for a new table; if consecutive data rows... [k,k1]> If the rows continue across pages, they will be merged into the same table; [k,k1]> If the rows are not part of the table, they will be merged into the table containing the last row of the previous page; the merged data will be a structured table; non-table content rows will not be merged. and These are the preset merging thresholds for data rows and cross-page rows, respectively; k is the data row currently being processed, and k1 is the data row being compared with the current row k.

9. The document table structure recognition method for enhanced retrieval of large-scale water conservancy models according to claim 4, characterized in that, The column area The horizontal projections of all physical objects do not intersect; The row area The vertical projections of all physical objects do not intersect.

Images