A table structure reconstruction method, device and medium based on a hybrid expert model

Abstract

The application discloses a table structure reconstruction method and device based on a hybrid expert model and a medium, and relates to the technical field of computer vision. The method comprises the following steps: performing region detection on an input table to obtain candidate regions of the input table, and generating a multi-scale feature map corresponding to the candidate regions; based on the multi-scale feature map, selecting an expert sub-model to be used in a preset hybrid expert model library through a gate network; calling the expert sub-model to be used, and analyzing the candidate regions to generate local table structure data; and performing content correlation reconstruction on the local table structure data to obtain a reconstructed table. Through the above method, the complex table is analyzed and reconstructed through the hybrid expert model and the gate network, and the accuracy of structure recognition and content extraction of the complex table is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, device and medium for reconstructing table structures based on a hybrid expert model. Background Technology

[0002] Most current mainstream table recognition technologies are based on deep learning models to automatically extract features. For example, convolutional neural networks are used for feature extraction, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) are used for sequence data processing, or models based on the Transformer architecture are used to achieve semantic-level understanding of table content. These technologies are usually used in combination in practical applications. Deep learning models are used to detect table regions, image processing techniques are used to extract the structure, and finally OCR technology is used to recognize cell content.

[0003] However, existing technologies still have many shortcomings when processing complex tables. First, their processing capabilities for complex table formats are limited, especially when faced with tables without obvious borders, rotated, or distorted, table recognition technology struggles to accurately identify table boundaries and structures. In addition, existing technologies lack real-time processing capabilities, making it difficult to meet the demands for efficient data processing in practical applications, such as rapid document processing in the logistics industry.

[0004] Therefore, how to parse and reconstruct the table structure for complex tabular data to obtain tables that conform to standard specifications has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a table structure reconstruction method, device, and medium based on a hybrid expert model to solve the following technical problem: how to parse and reconstruct the table structure for complex table data to obtain a table that conforms to standard specifications.

[0006] In a first aspect, embodiments of this application provide a table structure reconstruction method based on a hybrid expert model. The method includes: performing region detection on an input table to obtain candidate regions of the input table and generating multi-scale feature maps corresponding to the candidate regions; selecting an expert sub-model to be used from a preset hybrid expert model library through a gating network based on the multi-scale feature maps; calling the expert sub-model to be used to parse the candidate regions to generate local table structure data; and performing content association reconstruction on the local table structure data to obtain a reconstructed table.

[0007] In one implementation of this application, region detection is performed on the input table to obtain candidate regions of the input table, and multi-scale feature maps corresponding to the candidate regions are generated. Specifically, this includes: generating initial candidate regions on the input table based on an anchor frame structure; determining the confidence score of the initial candidate regions based on residual normalization scores, and removing redundant candidate regions from the initial candidate regions based on the confidence scores to obtain candidate regions of the input table; extracting shallow edge features and deep semantic features of the candidate regions through a convolutional backbone network, and fusing the shallow edge features and deep semantic features through a feature pyramid network to obtain multi-scale feature maps of the candidate regions.

[0008] In one implementation of this application, the method further includes: constructing a hybrid expert model, specifically including: acquiring sample table data; inputting the sample table data into a line detection algorithm and a three-branch network to be trained for training until a converged diagonal table header expert sub-model is obtained; and grouping the sample table data and performing row-column aligned distribution based on a preset vertical coordinate clustering algorithm to obtain a wireless table expert sub-model; and fusing the diagonal table header expert sub-model and the wireless table expert sub-model to obtain a hybrid expert model.

[0009] In one implementation of this application, sample table data is input into a line detection algorithm and a three-branch network to be trained until a converged diagonal header expert sub-model is obtained. Specifically, this includes: locating diagonal structures in the sample table data based on the line detection algorithm; the three-branch network includes a diagonal detection branch, a text localization branch, and an association modeling branch; wherein, the diagonal detection branch is used to output a pixel-level heatmap of the diagonal structure, the text localization branch is used to generate a first text block, and the association modeling branch is used to construct a spatial semantic mapping between the diagonal structure and the text block; the pixel-level heatmap, the first text block, and the spatial semantic mapping are associated to obtain the diagonal header expert sub-model.

[0010] In one implementation of this application, the sample table data is grouped and row-column aligned based on a preset vertical coordinate clustering algorithm to obtain a wireless table expert sub-model. Specifically, this includes: grouping the sample table data based on a preset vertical coordinate clustering algorithm to obtain a second text block; and aligning the logical relationships of the second text block based on a preset region adjacency graph algorithm to obtain a wireless table expert sub-model.

[0011] In one implementation of this application, based on multi-scale feature maps, an expert sub-model to be used is selected from a preset hybrid expert model library through a gating network. Specifically, this includes: extracting global context feature vectors from the multi-scale feature maps; inputting the global context feature vectors into the input layer of the gating network to obtain the probability distribution of the feature vectors through nonlinear transformation and normalization; and selecting the expert sub-model to be used from the hybrid expert model library based on the probability distribution.

[0012] In one implementation of this application, the expert sub-model to be used is invoked to parse the candidate region to generate local table structure data. Specifically, this includes: inputting the multi-scale feature map of the candidate region into the expert sub-model to be used and extracting special structures in the candidate region; and identifying and parsing the special structures through the expert sub-model to be used to obtain local table structure data.

[0013] In one implementation of this application, content association reconstruction is performed on local table structure data to obtain a reconstructed table. Specifically, this includes: verifying the topological relationship of the local table structure data and fixing logical conflicts in the topological relationship; and reconstructing the local table structure data based on the topological relationship to obtain a reconstructed table.

[0014] Secondly, embodiments of this application also provide a table structure reconstruction device based on a hybrid expert model. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: perform region detection on an input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions; select an expert sub-model to be used from a preset hybrid expert model library using a gating network based on the multi-scale feature maps; call the expert sub-model to be used to parse the candidate regions to generate local table structure data; and perform content association reconstruction on the local table structure data to obtain a reconstructed table.

[0015] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for table structure reconstruction based on a hybrid expert model, storing computer-executable instructions. The computer-executable instructions are configured to: perform region detection on the input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions; select an expert sub-model to be used from a preset hybrid expert model library through a gating network based on the multi-scale feature maps; call the expert sub-model to be used to parse the candidate regions to generate local table structure data; and perform content association reconstruction on the local table structure data to obtain the reconstructed table.

[0016] This application provides a table structure reconstruction method, device, and medium based on a hybrid expert model, which has the following advantages: It performs regional monitoring on the input table and generates multi-scale feature maps, simultaneously capturing the overall table framework and local cell details; it selects the expert sub-model to be used from a pre-set hybrid expert model library through a gating network, flexibly choosing the most suitable parsing model according to different table features and scenarios, avoiding the limitations of a single model when facing diverse tables; and it repairs the fragmentation problem that occurs during local parsing by reconstructing the content association of local table structure data, ensuring the integrity of the table's structure and content. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A flowchart of a table structure reconstruction method based on a hybrid expert model is provided for embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the internal structure of a table structure reconstruction device based on a hybrid expert model, provided as an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a table structure reconstruction method, device, and medium based on a hybrid expert model to solve the following technical problem: how to parse and reconstruct the table structure for complex table data to obtain a table that conforms to standard specifications.

[0022] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This document provides a flowchart of a table structure reconstruction method based on a hybrid expert model, as illustrated in an embodiment of this application. Figure 1 As shown in the figure, the table structure reconstruction method based on a hybrid expert model provided in this application specifically includes the following steps:

[0024] Step 10: Perform region detection on the input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions.

[0025] As an optional embodiment, region detection is performed on the input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions. Specifically, it may include: Step 101: Based on the anchor frame structure, generate initial candidate regions on the input table.

[0026] In this step, the anchor frame framework is a commonly used technical framework in object detection and image recognition tasks. Its core idea is to predefine a set of anchor points at different locations and scales in the input image. These anchor points can be regarded as potential target center points or key points. Based on these anchor points, a series of candidate regions are generated, which cover various locations and sizes in the image that may contain target objects. In the input table image or document, the operation based on the anchor frame framework treats the table as a series of target objects to be detected. By placing anchor points on the table image, these anchor points can cover different parts of the table, including the overall outline of the table, the boundaries of cells, and key areas such as the table header and footer. The initial candidate regions generated based on these anchor points are a preliminary estimate of the table structure or content areas that may exist in the table. These candidate regions will be further analyzed and filtered to determine whether they actually contain valid information of the table.

[0027] Step 102: Based on the residual normalization score, determine the confidence score of the initial candidate region, and based on the confidence score, remove redundant candidate regions from the initial candidate region to obtain the candidate regions for the input table.

[0028] In this step, residual normalization scoring is a method for evaluating candidate regions. In this application, residuals can be understood as the difference or error between a candidate region and the actual table region. By calculating the residual of each initial candidate region and normalizing it, a standardized score can be obtained. This score reflects the degree of matching between the candidate region and the actual table region. The normalization process is to make the scores of different candidate regions comparable and eliminate score differences caused by factors such as the size and position of the candidate region. Confidence score is an important indicator of whether a candidate region truly contains table information. A higher confidence score means that the candidate region is more likely to be a valid area of ​​the table, while a lower confidence score indicates that the candidate region may be noise or a misjudgment. Based on the confidence score obtained from residual normalization scoring, the initial candidate regions can be sorted and filtered. During the process of generating initial candidate regions based on the anchor frame structure, multiple candidate regions may overlap or be close. These redundant candidate regions increase the complexity and computational load of subsequent processing. Removing redundant candidate regions by using confidence scoring and retaining candidate regions with higher confidence can reduce the number of candidate regions in subsequent processing and improve the efficiency and accuracy of table parsing.

[0029] Step 103: Extract shallow edge features and deep semantic features of candidate regions through a convolutional backbone network, and fuse shallow edge features and deep semantic features through a feature pyramid network to obtain multi-scale feature maps of candidate regions.

[0030] In this step, the convolutional backbone network (CBR) is a deep convolutional neural network used to extract features from the input image or candidate regions. Through a series of convolutional layers, pooling layers, and activation functions, it can automatically learn the feature representation of the input data. In table parsing, the CBR can extract features from the candidate regions, including shallow edge features and deep semantic features. Shallow edge features mainly reflect low-level visual information in the candidate regions, such as edges and textures. These features are crucial for identifying table boundaries and cell outlines. For example, table borders and cell separators can be detected and extracted using shallow edge features. Deep semantic features... Features reflect higher-level semantic information in candidate regions, such as the content type and structural layout of the table. Deep semantic features can help the model understand the overall structure and semantic meaning of the table, for example, distinguishing the table header, body and footer, and identifying the data type (such as numbers, text, etc.) in the table. Feature pyramid network is a network structure for multi-scale feature fusion. It generates a feature pyramid containing multi-scale information by upsampling, downsampling and fusing features extracted from different levels by the convolutional backbone network. In this process, shallow edge features and deep semantic features are effectively combined, enabling the model to capture the detailed information and semantic information of the table at different scales at the same time.

[0031] Step 20: Based on the multi-scale feature map, select the expert sub-model to be used from the preset hybrid expert model library through the gating network.

[0032] As an optional embodiment, based on multi-scale feature maps, a gated network is used to select an expert sub-model to be used from a preset hybrid expert model library. Specifically, this may include: Step 201: Constructing a hybrid expert model.

[0033] As an optional implementation, constructing a hybrid expert model may specifically include: Step 2011: Obtaining sample table data. Step 2012: Inputting the sample table data into the line detection algorithm and the three-branch network to be trained for training until a converged diagonal table header expert sub-model is obtained.

[0034] As an optional embodiment, the sample table data is input into the line detection algorithm and the three-branch network to be trained until a converged diagonal table header expert sub-model is obtained. Specifically, it may include: Step 20121: Based on the line detection algorithm, locate the diagonal structure in the sample table data.

[0035] In this step, straight lines in a table are typically used to represent table borders, separators, or diagonal headers. By using a straight line detection algorithm, straight line features in the table can be extracted, providing basic information for subsequent table structure analysis. These straight line features are crucial for understanding the layout and structure of a table, especially when dealing with complex structures such as diagonal headers.

[0036] Step 20122: The three-branch network includes a diagonal line detection branch, a text localization branch, and an association modeling branch; among which, the diagonal line detection branch is used to output a pixel-level heatmap of the diagonal line structure, the text localization branch is used to generate the first text block, and the association modeling branch is used to construct a spatial semantic mapping between the diagonal line structure and the text block.

[0037] In this step, the three-branch network is a deep learning architecture specifically designed to handle diagonal header structures. It consists of three branches, each responsible for a different task: The diagonal detection branch analyzes the features of the input image and outputs a pixel-level heatmap of the diagonal structure. A heatmap is an intuitive representation where the value of each pixel represents the probability or intensity of the presence of a diagonal line at that location. In this way, the diagonal detection branch can accurately locate the position and shape of the diagonal line; The text localization branch generates the first text blocks using a text detection algorithm. These text blocks represent the position and boundaries of the text content in the table. The generation of the first text blocks provides the foundation for subsequent text content parsing and semantic association; The association modeling branch analyzes the spatial relationship between the diagonal structure and the text blocks to determine the connection relationship and semantic role between the diagonal lines and the text blocks. For example, the association modeling branch can identify which text blocks are connected by diagonal lines and the semantic hierarchy of these text blocks in the table (such as main header, sub-header, etc.).

[0038] Step 20123: Associate the pixel-level heatmap, the first text block, and the spatial semantic mapping to obtain the expert sub-model of the diagonal header.

[0039] In this step, by inputting sample table data into a three-branch network for training, the network can learn the feature representation and parsing rules of the diagonal table header. During the training process, the network parameters are continuously adjusted to minimize the error between the prediction results and the true labels until the network converges, thus obtaining an expert sub-model that can accurately identify and parse the diagonal table header.

[0040] Step 2013: Based on the preset vertical coordinate clustering algorithm, the sample table data is grouped and distributed with row and column alignment to obtain the wireless table expert sub-model.

[0041] As an optional embodiment, the sample table data is grouped and row-column aligned based on a preset vertical coordinate clustering algorithm to obtain a wireless table expert sub-model. Specifically, it may include: Step 20131: Grouping the sample table data based on a preset vertical coordinate clustering algorithm to obtain a second text block.

[0042] In this step, the structure of the wireless table is mainly reflected through the arrangement and alignment of text, rather than through explicit border lines. In this case, the vertical coordinate clustering algorithm clusters the text blocks in the table by using their vertical coordinates to group them into different rows. The vertical coordinates of the text blocks refer to the vertical position information of the text in the table. Through the clustering algorithm, text blocks with similar vertical positions can be grouped together, thereby identifying the row structure of the table. This can effectively handle the text arrangement problem in wireless tables and accurately identify the row structure of the table even without explicit border lines.

[0043] Step 20132: Based on the preset region adjacency graph algorithm, align the logical relationships of the second text block to obtain the wireless table expert sub-model.

[0044] In this step, the region adjacency graph algorithm is a technique used to analyze and model the spatial relationships between text blocks. In the wireless table, although the text blocks have been grouped into rows by the vertical coordinate clustering algorithm, it is still necessary to further determine the column alignment relationships between these rows. The region adjacency graph algorithm constructs a graph structure in which each node represents a text block and the edges represent the adjacency relationships between text blocks. By analyzing these adjacency relationships, it can determine which text blocks are aligned in the horizontal direction, thereby identifying the column structure of the table. This algorithm can handle complex spatial relationships between text blocks, such as handling problems such as overlap, gaps, or irregular arrangements between text blocks. In this way, the column structure of the table can be accurately identified.

[0045] Furthermore, by aligning the logical relationships of the second text block using the region adjacency graph algorithm, a specialized expert sub-model for processing wireless tables can be obtained. This sub-model can effectively parse the structure of wireless tables, identify the rows and columns of the table, and correctly allocate the text content to the corresponding cells. The design of the wireless table expert sub-model enables the table parsing engine to process document content that does not have explicit borders but still has a table structure. It can adapt to various complex wireless table layouts, improve the robustness and accuracy of table parsing, and provide strong technical support for intelligent document processing.

[0046] Step 2014: Merge the expert sub-model of the diagonal header table and the expert sub-model of the wireless table to obtain a hybrid expert model.

[0047] In this step, the expert sub-model for diagonal headers and the expert sub-model for wireless tables were optimized for different types of table structures. However, tables in real-world applications are often diverse and complex, and may contain features of both diagonal headers and wireless tables. Therefore, it is necessary to merge these two expert sub-models to build a hybrid expert model that can handle multiple table structures.

[0048] Step 202: Extract the global context feature vector from the multi-scale feature map.

[0049] In this step, feature maps of different scales are concatenated along the channel dimension. A convolutional network is then used to perform weighted fusion of the feature maps of different scales, compressing the information in the multi-scale feature maps into a fixed-length vector, thus obtaining the global context feature vector. The purpose of the global context feature vector is to integrate the information in the multi-scale feature maps into a compact feature representation. This feature vector can reflect the global features of the entire input data, including the overall structure and layout of the table and the relationships between different parts. It can reflect the overall features of the entire input data and provides an important basis for the subsequent gating network to select expert sub-models.

[0050] Step 203: Input the global context feature vector into the input layer of the gated network to obtain the probability distribution of the feature vector through nonlinear transformation and normalization.

[0051] In this step, the gating network is a network structure used to dynamically select expert sub-models. Its core function is to determine the type of table structure of the input data based on its features and select the most suitable expert sub-model for processing. By learning the features of the input data, the gating network can automatically identify the type and complexity of the input data, thereby selecting the most suitable expert sub-model for each input data. The global context feature vector is processed by an activation function to introduce non-linear characteristics. This non-linear transformation allows the gating network to learn more complex feature representations, thus better capturing the feature differences in the input data. Normalization involves adjusting the values ​​of the feature vectors to a uniform range, ensuring consistency in the numerical ranges of different features and preventing certain features from having an unreasonable dominant effect on the final result due to excessively large or small numerical ranges. Through the processing of the gating network, the global context feature vector is mapped to a probability space. The probability distribution represents the likelihood that the input data belongs to the processing range of different expert sub-models. For example, if the input data is a complex table with a diagonal header, the gating network may output a higher probability value, indicating that the input data is more likely to be processed by the expert sub-model with the diagonal header.

[0052] Step 204: Based on the probability distribution, select the expert sub-models to be used from the hybrid expert model library.

[0053] In this step, based on the probability distribution, the gating network can dynamically select one or more expert sub-models to process the input data, based on the highest value of the probability distribution or the Top-k strategy, that is, selecting the expert sub-model with the highest probability or selecting the top k expert sub-models with the highest probability.

[0054] Step 30: Call the expert sub-model to be used to parse the candidate regions and generate local table structure data.

[0055] As an optional implementation, the expert sub-model to be used is invoked to parse the candidate region to generate local table structure data. Specifically, it may include: Step 301: Input the multi-scale feature map of the candidate region into the expert sub-model to be used, and extract the special structure in the candidate region.

[0056] In this step, the multi-scale feature map of the candidate region is input into the expert sub-model to be used, so that the expert sub-model can understand the features of the candidate region from multiple perspectives and extract the parts of the candidate region with specific features or complex layouts, such as diagonal headers, merged cells, nested tables, etc., as special results, which are convenient for subsequent special processing.

[0057] Step 302: Identify and parse the special structure using the expert sub-model to be used to obtain local table structure data.

[0058] In this step, after identifying special structures, the expert sub-model further analyzes the semantic information of these structures. For example, the expert sub-model for diagonal headers not only identifies the position of the diagonal lines but also analyzes the semantic relationship between the diagonal lines and text blocks to determine the hierarchical structure of the header; the expert sub-model for wireless tables analyzes the alignment relationship of text blocks to generate the row and column structure of the table; after analysis by the expert sub-models, structured data about the candidate regions is generated, which is the local table structure data. This data includes the row and column structure of the table, cell content, semantic information of special structures, etc. The local table structure data is an intermediate result in the table parsing process and will be integrated by the content association reconstruction module to generate the complete table structure. The accuracy and completeness of the local table structure data directly affect the quality of the final table parsing.

[0059] Step 40: Reconstruct the content association of the local table structure data to obtain the reconstructed table.

[0060] As an optional embodiment, content association reconstruction is performed on the local table structure data to obtain a reconstructed table, which may specifically include: Step 401: verify the topological relationship of the local table structure data and fix the logical conflicts in the topological relationship.

[0061] In this step, the semantic information of the local table structure data, such as row and column structure, cell content, and special structures (e.g., diagonal headers, merged cells), is spatially and logically connected to obtain the topological relationship of the local table structure data. This is then verified to ensure that the connections between these structural information are reasonable and consistent. For example, there should be clear alignment between rows and columns, the merging of cells should be logically reasonable, and the connection between diagonal headers and text blocks should be correct. Logical conflicts should be avoided to prevent them from affecting the accuracy and completeness of table parsing. The topological relationships in the local table structure data are checked for logical consistency. These rules may include row and column alignment rules, cell merging rules, and header-body connection rules. The verification process will detect logical conflicts. Based on the detected conflicts, corresponding remedial measures are taken, including adjusting cell merging relationships, realigning row and column structures, and correcting diagonal header connections. The purpose of this remediation is to ensure that the topological relationships of the local table structure data conform to the table's logical rules, guaranteeing the rationality and consistency of the table structure.

[0062] Step 402: Based on the topological relationship, reconstruct the local table structure data to obtain the reconstructed table.

[0063] In this step, reconstruction refers to integrating and optimizing the local table structure data to generate a complete and accurate table structure. During the table parsing process, the local table structure data may come from different expert sub-models, and these data may be fragmented or inconsistent. Through reconstruction, these local data can be integrated into a complete table structure. The reconstruction process needs to be based on topological relationships, which define the connections and hierarchical relationships between the various parts of the table structure. Only by correctly understanding and processing these relationships can a logically reasonable table structure be generated.

[0064] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a table structure reconstruction device based on a hybrid expert model, the structure of which is as follows: Figure 2 As shown.

[0065] Figure 2 This is a schematic diagram illustrating the internal structure of a table-based reconstruction device using a hybrid expert model, as provided in an embodiment of this application. Figure 2 As shown, the device includes:

[0066] At least one processor 201;

[0067] And a memory 202 that is communicatively connected to at least one processor;

[0068] The memory 202 stores instructions executable by at least one processor. These instructions are executed by at least one processor 201 to enable the at least one processor 201 to: perform region detection on the input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions; select an expert sub-model to be used from a preset hybrid expert model library through a gating network based on the multi-scale feature maps; call the expert sub-model to be used to parse the candidate regions to generate local table structure data; and perform content association reconstruction on the local table structure data to obtain a reconstructed table.

[0069] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for table structure reconstruction based on a hybrid expert model stores computer-executable instructions. The computer-executable instructions are configured to: perform region detection on the input table to obtain candidate regions of the input table and generate multi-scale feature maps corresponding to the candidate regions; select an expert sub-model to be used from a preset hybrid expert model library through a gating network based on the multi-scale feature maps; call the expert sub-model to be used to parse the candidate regions to generate local table structure data; and perform content association reconstruction on the local table structure data to obtain the reconstructed table.

[0070] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0071] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A table structure reconstruction method based on a hybrid expert model, characterized in that, The method includes: Region detection is performed on the input table to obtain candidate regions of the input table, and multi-scale feature maps corresponding to the candidate regions are generated. Based on the multi-scale feature map, a gated network selects an expert sub-model to be used from a preset hybrid expert model library; wherein, the expert sub-model to be used is used to process a specific table structure type; The expert sub-model to be used is invoked to parse the candidate region in order to generate local table structure data; The local table structure data is reconstructed by content association to obtain a reconstructed table; Constructing a hybrid expert model specifically includes: Obtain sample table data; The sample table data is input into the line detection algorithm and the three-branch network to be trained until a converged expert sub-model of the diagonal table header is obtained; and Based on a preset vertical coordinate clustering algorithm, the sample table data is grouped and distributed with row and column alignment to obtain a wireless table expert sub-model. The expert sub-model of the diagonal header and the expert sub-model of the wireless table are fused to obtain the hybrid expert model; The sample table data is input into the line detection algorithm and the three-branch network to be trained until a converged expert sub-model of the diagonal table header is obtained, specifically including: Based on the line detection algorithm, the diagonal line structure in the sample table data is located; The three-branch network includes a diagonal line detection branch, a text localization branch, and an association modeling branch; wherein, the diagonal line detection branch is used to output a pixel-level heatmap of the diagonal line structure, the text localization branch is used to generate a first text block, and the association modeling branch is used to construct a spatial semantic mapping between the diagonal line structure and the text block; The pixel-level heatmap, the first text block, and the spatial semantic mapping are associated to obtain the expert sub-model of the diagonal header; Based on a preset vertical coordinate clustering algorithm, the sample table data is grouped and distributed with row and column alignment to obtain a wireless table expert sub-model, specifically including: Based on a preset vertical coordinate clustering algorithm, the sample table data is grouped to obtain the second text block; Based on a preset regional adjacency graph algorithm, the logical relationships of the second text block are aligned to obtain the wireless table expert sub-model.

2. The table structure reconstruction method based on a hybrid expert model according to claim 1, characterized in that, The input table is subjected to region detection to obtain candidate regions of the input table, and multi-scale feature maps corresponding to the candidate regions are generated, specifically including: Based on the anchor frame structure, an initial candidate region is generated on the input table; Based on the residual normalized score, the confidence score of the initial candidate region is determined, and based on the confidence score, redundant candidate regions in the initial candidate region are removed to obtain the candidate regions of the input table. The shallow edge features and deep semantic features of the candidate region are extracted by a convolutional backbone network, and the shallow edge features and deep semantic features are fused by a feature pyramid network to obtain a multi-scale feature map of the candidate region.

3. The table structure reconstruction method based on a hybrid expert model according to claim 1, characterized in that, Based on the multi-scale feature maps, a gating network is used to select an expert sub-model from a pre-defined hybrid expert model library, specifically including: Extract global context feature vectors from the multi-scale feature map; The global context feature vector is input into the input layer of the gated network to obtain the probability distribution of the feature vector through nonlinear transformation and normalization. Based on the probability distribution, the expert sub-model to be used is selected from the hybrid expert model library.

4. The table structure reconstruction method based on a hybrid expert model according to claim 1, characterized in that, The expert sub-model to be used is invoked to parse the candidate regions and generate local table structure data, specifically including: The multi-scale feature map of the candidate region is input into the expert sub-model to be used, and the special structures in the candidate region are extracted. The special structure is identified and parsed using the expert sub-model to be used, so as to obtain the local table structure data.

5. The table structure reconstruction method based on a hybrid expert model according to claim 1, characterized in that, The local table structure data is reconstructed by content association to obtain a reconstructed table, specifically including: Perform topological relationship verification on the local table structure data and repair logical conflicts in the topological relationships; Based on the aforementioned topological relationships, the local table structure data is reconstructed to obtain a reconstructed table.

6. A table structure reconstruction device based on a hybrid expert model, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a method as described in any one of claims 1-5.

7. A non-volatile computer storage medium based on a table structure reconstruction using a hybrid expert model, storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement a method as described in any one of claims 1-5.

Images