Visual-based Cell Structure Recognition Using Hierarchical Neural Networks and Clustering from Cell Boundaries to Structures

Abstract

Provided herein are methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundary-to-structure clustering. The computer-implemented method includes detecting a style of a given table using at least one style classification model, selecting a cell detection model appropriate for the detected style based at least in part on the detected style, detecting cells in the given table using the selected cell detection model, and outputting, to at least one user, information about the detected cells, including image coordinates of one or more bounding boxes associated with the detected cells.

Description

Technical Field

[0001] This application generally relates to information technology, and more specifically to document analysis technology.

Background Art

[0002] Analysis and understanding of documents typically require accurate recognition of the cell structure of table content. However, many documents are stored and distributed in Portable Document Format (PDF) or image format, which often fails to preserve the format of any tables included in the document. Furthermore, conventional document analysis techniques generally rely on manual features and require expensive and time-consuming human expertise when encountering new document styles. Also, conventional object detection techniques often face challenges in detecting small or horizontally elongated or both types of objects such as table cells, and typically do not consider visual elements of the document such as boundaries, shading, and fonts.

Summary of the Invention

[0003] In one embodiment of the present invention, a technique for visual-based cell structure recognition using a hierarchical neural network and cell boundaries to structure clustering is provided. An exemplary computer-implemented method includes detecting the style of a given table using at least one style classification model and selecting a cell detection model suitable for the detected style based at least in part on the detected style. The method further includes using the selected cell detection model to detect cells within a given table and outputting information about the detected cells, including the image coordinates of one or more bounding boxes associated with the detected cells, to at least one user.

[0004] In another embodiment of the present invention, an exemplary computer-implemented method can include removing one or more cell boxes associated with cells within a table that do not overlap with any text box associated with the table, and expanding each of the one or more remaining cell boxes associated with cells within the table until each of the one or more remaining cell boxes is expanded to a maximum horizontal width without overlapping with one or more other remaining cell boxes. The method can further include sampling horizontally and vertically at the center of each expanded cell box to determine the number of rows within the table and the number of columns within the table, and determining an alignment with respect to the rows and columns of the table based at least in part on the one or more remaining cell boxes prior to the expanding. Further, the method can include using at least one K-means clustering technique on the one or more remaining cell boxes based at least in part on the determined number of rows within the table and the determined number of columns within the table, and assigning each of the one or more remaining cell boxes to respective rows and respective columns based at least in part on the determined alignment.

[0005] Another embodiment of the present invention or an element thereof can be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when executed, cause a computer to perform a plurality of method steps described herein. Further, another embodiment of the present invention or an element thereof can be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform the method steps. Further, another embodiment of the present invention or an element thereof can be implemented in the form of means for performing the method steps described herein or an element thereof, the means including hardware modules (singular or plural) or a combination of hardware and software modules, the software modules being stored in a tangible computer-readable storage medium (or plural such media).

[0006] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of its exemplary embodiments, which should be read in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0007] Next, embodiments of the present invention will be described by way of example only, with reference to the accompanying drawings.

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0009] As described herein, one embodiment of the present invention includes vision-based cell structure recognition using a hierarchical neural network and cell boundary to structure clustering. Such embodiments include using one or more neural networks to identify cells (of a table) and using at least one cell clustering technique to determine the table structure.

[0010] Furthermore, at least one embodiment includes generating or implementing or both a Global Table Extractor (GTE), a visual guidance framework for cell structure recognition, which can be built, for example, on top of one or more object detection models. Such embodiments can further include implementing at least one algorithm for detecting the boundaries of one or more cells using a style recognition hierarchical model, and such detected boundaries can be transformed into structural details using alignment and coordinate-based clustering.

[0011] As described in more detail herein, after cells are detected, one or more embodiments include leveraging sampling and clustering methods to infer the number of rows and columns in the table and the location of each cell in the table. Such clustering methods also identify table cells that span multiple rows and columns.

[0012] FIG. 1 is a diagram showing a system architecture according to an embodiment of the present invention. For illustration purposes, FIG. 1 shows an overview of hierarchical cell detection in one or more embodiments, which embodiments include, for example, an input of at least one full-page image, from which one or more tables 120 are detected, which are processed by a GTE cell component 122 to generate a cell boundary output 136.

[0013] As described in more detail herein, at least one embodiment includes generating or implementing or both a GTE framework (GTE cell component 122) that includes a network for table boundary detection and cell boundary detection. As shown in the example of FIG. 1, the input to the GTE cell component 122 can include the image format of the document page, while the GTE cell component 122 also depends on the table boundaries (determined with respect to the detected table 120) to generate the cell structure for each particular table.

[0014] Referring back to FIG. 1, with respect to the GTE cell component 122, it should be recognized that the table often conforms to the global style that determines the rules and meaning of that component. For example, there are some tables with visible vertical and horizontal gridlines for all rows and columns that easily define cell boundaries. However, there are other styles that have no gridlines or only intermittent breaks. In such cases, a model that looks only at the local surroundings will not be able to determine whether the gridlines represent the start of a new cell.

[0015] Accordingly, at least one embodiment includes training at least one attribute neural network (e.g., network 126) for the purpose of classifying the presence of vertical graph gridlines within a table. As shown in the example of FIG. 1, an input in the form of an entire page with a table mask 124 (generated at least in part based on the input image and the detected table 120) is processed by the attribute neural network 126. The output of the neural network 126 determines which of the two cell detection networks is used. The cell network 132, generalized for all table styles, can be trained on data consisting of tables with and without graph lines and / or data adding one or more vertical and / or horizontal boundaries at the midpoints between cells (as shown, for example, via component 134). The cell network 128, specialized for tables with graph lines, is trained on complete boundary data in addition to the initial input and generates an output (such as component 130) where the network depends on the graph lines of the table when detecting cell boundaries.

[0016] To convert the cell - bounding - box output into a logical structure (e.g., component 136), one or more embodiments include aligning the cell boxes to text lines as extracted from a document (e.g., a PDF document). Subsequently, the embodiments include determining the number of rows and columns respectively by vertical and horizontal sampling. Prior to vertical sampling to determine the number of rows, at least one embodiment can include expanding the left and right ends of the cells without overlapping existing cells to account for rows with missing cells. Then, the embodiments can include inferring the vertical and horizontal alignment of the table such that the edges of the cell boxes are best aligned with one or more other cells. At least one embodiment includes determining the row and column positions using K - means clustering on the cell - bounding - box coordinates. Subsequently, one or more embodiments include assigning the row and column positions to each cell based on their box positions and merging cells when necessary.

[0017] Furthermore, at least one embodiment includes leveraging the assumption that cell content generally starts with a capital letter. Thus, a cell starting with a lower - case letter is determined to be a case of over - splitting, and such a cell is merged with an adjacent cell (e.g., the cell above). Additionally, one or more embodiments include performing at least one post - processing step that can include assigning positions to the remaining text boxes that did not overlap with any of the detected cells and splitting cells in certain cases where there are gaps nearby. Before generating the final logical structure 136 for each cell in the table, at least one embodiment can include expanding the row and column ranges of the cells when an adjacent empty row or column intersects with the text box, as this is likely to be a hierarchical cell spanning multiple rows and columns.

[0018] Figure 2 shows an exemplary code snippet for a Cell Boundary to Structure Cluster Algorithm according to an exemplary embodiment of the present invention. In this embodiment, the exemplary code snippet 200 is executed by at least one processing system or device or both, or under its control. For example, the exemplary code snippet 200 can be considered to include a portion of the software implementation of at least a part of the GTE cell component 122 of the embodiment of FIG. 1.

[0019] The exemplary code snippet 200 shows the preprocess of cell bounding boxes that can be merged when the boundary boxes intersect sufficiently and can be deleted when there are no overlapping text boxes. The exemplary code snippet 200 further shows the process of Assign Cell Row and Column Location, which assigns the row and column positions of the cell to each cell by determining the number of rows and columns and clustering to find the boundaries of the determined rows and columns.

[0020] It should be recognized that this particular exemplary code snippet shows at least one exemplary implementation of a portion of the cluster algorithm from cell boundary to structure, and alternative implementations of the process can be used in other embodiments.

[0021] Figure 3 is a flowchart showing a technique (e.g., a computer-implemented method for use with a given table in a document) according to an embodiment of the present invention. Step 302 includes detecting the style of a given table using at least one style classification model. In one or more embodiments, the at least one style classification model includes a deep neural network trained on a plurality of tables including a plurality of formatting attributes (e.g., attributes related to grid lines, emphasis, bolding, font size, font type, italic, etc.). Further, in at least one embodiment, detecting the style of a given table includes detecting the boundaries of the given table using one or more object detection models.

[0022] Step 304 includes selecting a cell detection model suitable for the detected style, at least partially based on the detected style. Step 306 includes detecting cells within the given table using the selected cell detection model. In at least one embodiment, the selected cell detection model is trained using at least one deep neural network on at least one table including at least one style similar to the style of the given table. Further, or alternatively, in such an embodiment, the selected cell detection model is trained using at least one deep neural network on a table including a plurality of styles.

[0023] Further, in one or more embodiments, detecting cells within a given table includes using one or more optical character recognition techniques in conjunction with the selected cell detection model.

[0024] Step 308 includes outputting information about the detected cells, including the image coordinates of one or more bounding boxes associated with the detected cells, to at least one user. The method shown in FIG. 3 can further include converting at least a portion of the one or more bounding boxes into a logical structure. In one or more embodiments, converting includes aligning at least a portion of the one or more bounding boxes with one or more text lines.

[0025] Furthermore, additional embodiments of the present invention include removing one or more cell boxes associated with cells within a table that do not overlap with any text boxes associated with the table, and expanding one or more remaining cell boxes associated with cells within the table until each of the one or more remaining cell boxes is expanded to its maximum horizontal width without overlapping with one or more other remaining cell boxes. Expanding one or more remaining cell boxes associated with the table can include, for example, expanding the one or more remaining cell boxes one by one and in the order from left to right and from top to bottom. Such embodiments further include sampling horizontally and vertically at the center of each expanded cell box to determine the number of rows and the number of columns of the table, and determining the alignment of the rows and columns of the table based at least in part on the one or more remaining cell boxes before the above expansion. Furthermore, such embodiments include using at least one K-means clustering technique for the one or more remaining cell boxes based at least in part on the determined number of rows and the determined number of columns within the table, and assigning each of the one or more remaining cell boxes to its respective row and its respective column based at least in part on the determined alignment.

[0026] Such embodiments further include expanding one or more cells into one or more adjacent empty cells when it is determined that a text portion of one or more cells overlaps one or more adjacent empty cells. Additionally or alternatively, such embodiments include splitting one or more multi-text line cells when it is determined that at least one adjacent empty cell exists, and reassigning corresponding text portions of the one or more split cells based at least in part on corresponding text positions.

[0027] The technique shown in FIG. 3 further includes preparing a system as described herein, where the system includes separate software modules, each of the separate software modules being embodied on a tangible computer-readable recordable storage medium. All of the modules (or any subset thereof) can be present on the same medium, or each can be present on, for example, different media. These modules can include any or all of the components shown in the figures or described herein or both. In one embodiment of the invention, the modules can be executed, for example, on a hardware processor. Next, method steps are executed using separate software modules of the system that are executed on the hardware processor as described above. Further, a computer program product includes a tangible computer-readable recordable storage medium having code adapted to be executed to perform at least one method step described herein, including preparing a system having separate software modules.

[0028] Furthermore, the technique shown in FIG. 3 can be implemented via a computer program product that can include computer-usable program code stored in a computer-readable storage medium within a data processing system, where the computer-usable program code is downloaded over a network from a remote data processing system. Also, in one embodiment of the present invention, the computer program product can include computer-usable program code stored in a computer-readable storage medium within a server data processing system, where the computer-usable program code is downloaded over a network to the remote data processing system for use within the computer-readable storage medium using the remote system.

[0029] One embodiment of the present invention or an element thereof can be implemented in the form of an apparatus that includes a memory and at least one processor coupled to the memory and is configured to perform exemplary method steps.

[0030] Furthermore, an embodiment of the present invention can utilize software executed on a computer or a workstation. Referring to FIG. 4, such an embodiment can use, for example, a processor 402, a memory 404, and an input / output interface formed by, for example, a display 406 and a keyboard 408. The term "processor" as used herein is intended to include any processing device, such as a CPU (Central Processing Unit) or other forms of processing circuitry or both. Further, the term "processor" can refer to more than one individual processor. The term "memory" is intended to include memory associated with the processor or CPU, such as RAM (Random Access Memory), ROM (Read Only Memory), fixed memory devices (e.g., hard drives), removable memory devices (e.g., floppy disks), flash memory, etc. Further, the phrase "input / output interface" as used herein is intended to include, for example, a mechanism for inputting data to the processing unit (e.g., a mouse), and a mechanism for providing results associated with the processing unit (e.g., a printer). The processor 402, the memory 404, and the input / output interface such as the display 406 and the keyboard 408 can be interconnected via, for example, a bus 410 as part of a data processing unit 412. Appropriate interconnections via, for example, the bus 410 can be provided to a network interface 414, such as a network card that can provide an interface to a computer network, and to a media interface 416, such as a floppy disk or CD-ROM drive that can provide an interface to a medium 418.

[0031] Accordingly, software containing instructions or code for performing the method of the present invention described herein can be stored in a relevant memory device (e.g., ROM, fixed or removable memory), and when it is ready for use, it can be partially or fully loaded (e.g., into RAM) and implemented by a CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0032] A data processing system suitable for storing or executing program code or both will include at least one processor 402 directly or indirectly coupled to memory 404 through system bus 410. Memory elements can include local memory used during actual execution of program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times code needs to be retrieved from bulk storage during execution.

[0033] Input / output or I / O devices (including, but not limited to, keyboard 408, display 406, pointing devices, etc.) can be coupled to the system directly (such as via bus 410) or through an intervening I / O controller (omitted for clarity).

[0034] A network adapter such as network interface 414 can be further connected to the system so that the data processing system can be coupled to other data processing systems or remote printers or storage devices through an intervening private or public network. Modems, cable modems, and Ethernet cards are some of the currently available types of network adapters.

[0035] As used in this specification, including the claims, "server" includes a physical data processing system (e.g., system 412 shown in FIG. 4) that executes a server program. It should be understood that such a physical server may or may not include a display and a keyboard.

[0036] The present invention can be integrated in a system, a method, a computer program product, or a combination thereof, at any possible technical detail level. The computer program product can include a computer-readable storage medium (s) having computer-readable program instructions for causing a processor to execute aspects of the present invention.

[0037] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following, namely, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device such as a raised structure in a groove in which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not construed as being a transient signal per se, such as a radio wave, or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0038] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or to an external computer or external storage device via a network, such as, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0039] The computer-readable program instructions for carrying out the operations of the present invention may be source code or object code described in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, may be executed partly on the user's computer and partly as a stand-alone software package, may be executed partly on the user's computer and partly on a remote computer, or may be executed entirely on the remote computer or server. In the last scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions in order to implement aspects of the present invention and to customize the electronic circuit.

[0040] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0041] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both. These computer program instructions can also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium contain instructions for implementing the aspects of the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both, including a product.

[0042] The computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both.

[0043] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in accordance with the functionality involved, actually be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams or flowchart diagrams, or combinations of blocks in the block diagrams or flowchart diagrams or both, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0044] Any of the methods described herein may include additional steps of providing a system that includes different software modules implemented on a computer-readable storage medium, which modules may include, for example, any or all of the components detailed herein. Next, the method steps can be executed using different software modules or sub-modules of the system, or both, as described above, executed on a hardware processor 402. Further, a computer program product can include a computer-readable storage medium having code adapted to be implemented to perform at least one method step described herein, including providing a system with different software modules.

[0045] In any case, it should be understood that the components described in this specification can be implemented in various forms of hardware, software, or a combination thereof, such as application-specific integrated circuits (ASICs), functional circuits, appropriately programmed digital computers with associated memories, and the like. Given the technology of the present invention provided herein, those skilled in the art will be able to consider other implementations of the components of the present invention.

[0046] Furthermore, it is to be understood in advance that the implementation of the teachings presented herein is not limited to a particular computing environment. Rather, embodiments of the present invention can be implemented with any type of computing environment now known or later developed.

[0047] For example, cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0048] The characteristics are as follows. On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capabilities such as server time and network storage as needed, without the need for a human to interact with the service provider. Broad network access: The capabilities are available over the network and are accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (such as mobile phones, laptops, and PDAs). Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and re-assigned according to demand. Consumers are generally location-independent in that they have no control or knowledge over the exact location of the resources provided, although they may be able to specify a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: Capabilities can be provisioned quickly and elastically, and in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time. Measured service: The cloud system automatically controls and optimizes resource use by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

[0049] The service model is as follows. Software as a Service (SaaS): The capabilities provided to the consumer are to use the provider's applications running on the cloud infrastructure. These applications are accessible from various client devices through a client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application capabilities, with the exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The function provided to the consumer is to deploy the applications created or obtained by the consumer, which are created using the programming languages and tools supported by the provider, onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, or storage, but controls the deployed applications and, in some cases, the environmental configuration that hosts the applications. Infrastructure as a Service (IaaS): The function provided to the consumer is to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software that may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has limited control over the operating system, storage, control of the deployed applications, and, in some cases, selection of network components (e.g., host firewalls).

[0050] The deployment model is as follows. Private cloud: The cloud infrastructure is operated solely for an organization. This can be managed by the organization or a third party and can exist on-premises or off-premises. Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). This can be managed by those organizations or a third party and can exist on-premises or off-premises. Public Cloud: The cloud infrastructure is available for use by the general public or a large industry group and is owned by an organization that sells cloud services. Hybrid Cloud: The cloud infrastructure remains a distinct entity but is a hybrid of two or more clouds (private, community, or public) that are linked together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0051] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is infrastructure that includes a network of interconnected nodes.

[0052] Referring now to FIG. 5, an exemplary cloud computing environment 50 is shown. As illustrated, cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices, such as, for example, a portable information terminal (PDA) or cellular telephone 54A, desktop computer 54B, laptop computer 54C, or automotive computer system 54N or a combination thereof, utilized by cloud consumers may communicate. The nodes 10 can communicate with one another. The nodes 10 can be physically or virtually grouped in one or more networks (not shown) such as the private cloud, community cloud, public cloud, or hybrid cloud described above, or a combination thereof. This enables cloud computing environment 50 to provide Infrastructure as a Service, Platform as a Service or Software as a Service, or a combination thereof, in which cloud consumers need not maintain resources on local computing devices. The types of computing devices 54A - N shown in FIG. 7 are merely intended to be exemplary, and it is understood that cloud computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network or network addressable connection, or both (e.g., using a web browser).

[0053] Referring now to FIG. 6, a set of functional abstractions provided by cloud computing environment 50 (FIG. 5) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 6 are merely intended to be exemplary and embodiments of the invention are not limited thereto. As illustrated, the following layers and corresponding functions are provided.

[0054] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based server 62, server 63, blade server 64, storage device 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0055] The virtualization layer 70 provides an abstraction layer that can provide the following examples of virtual entities: virtual server 71, virtual storage 72, virtual network 73 including a virtual private network, virtual applications and operating systems 74, and virtual client 75. In one example, the management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to execute tasks within a cloud computing environment. Metering and pricing 82 provides expense tracking when resources are utilized within a cloud computing environment and billing or charging for consumption of these resources.

[0056] In one example, these resources may include application software licenses. Security provides identification verification for cloud consumers and tasks, as well as protection for data and other resources. The user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that the required service levels are met. Planning and fulfillment of a service level agreement (SLA) 85 provides pre-arrangement and procurement of cloud computing resources whose future requirements are predicted according to the SLA.

[0057] The workload layer 90 provides examples of functions that can utilize the cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91, software development and life cycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and cell structure recognition 96 according to one or more of the present invention.

[0058] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, the term "comprise" or "comprising" or both, as used herein, while indicating the presence of the described features, integers, steps, operations, elements, or components or combinations thereof, do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof.

[0059] At least one embodiment of the present invention can provide advantageous effects such as generating a visual guidance framework for cell structure recognition that can be built on, for example, one or more object detection models.

[0060] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes, but they are not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, the practical application, or the technological improvements over the technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for recognizing a cell structure of a given table in a document by computer information processing, comprising: detecting a style of the given table using at least one style classification model including a deep neural network trained for a plurality of tables including a plurality of formatting attributes; selecting a cell detection model that is either a cell network generalized for all table styles suitable for the detected style or a cell network specialized for a table with graph lines, based at least in part on the pre-detected style; detecting cells within the given table using the selected cell detection model; outputting information about the pre-detected cells, including image coordinates of one or more bounding boxes associated with the detected cells, to at least one user; converting at least a part of the one or more bounding boxes into a logical structure, including aligning the at least a part of the one or more bounding boxes with one or more text lines; including The method is executed by at least one computing device. Method.

2. The method according to claim 1, wherein the plurality of formatting attributes includes attributes related to at least two of graph lines, emphasis, bolding, font size, font type, and italic.

3. The method according to claim 1, wherein the selected cell detection model is trained for tables including a plurality of styles using at least one deep neural network.

4. The method according to claim 1, wherein detecting the cells within the given table includes using one or more optical character recognition techniques together with the selected cell detection model.

5. A computer program including program instructions for causing a computer to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium storing the computer program according to claim 5.

7. A memory, operatively coupled to the memory, Using at least one style classification model including a deep neural network trained on multiple tables including multiple formatting attributes, detect the style of a given table, Based at least in part on the detected style, select a cell detection model that is either a cell network generalized for all table styles suitable for the detected style or a cell network specialized for tables with graph lines, Using the selected cell detection model, detect cells within the given table, Output information about the detected cells, including the image coordinates of one or more bounding boxes associated with the detected cells, to at least one user, Converting at least a portion of the one or more bounding boxes into a logical structure, including aligning the at least a portion of the one or more bounding boxes with one or more text lines, the converting At least one processor configured to A system including.

8. The system according to claim 7, wherein the plurality of formatting attributes includes attributes related to at least two of graph lines, emphasis, bolding, font size, font type, and italic.

9. The system according to claim 7, wherein the selected cell detection model is trained using one or more of at least one deep neural network for tables including multiple styles.

Images