Table recognition method and system

Abstract

The table recognition method and system provided in the specification acquires a target table image containing a target table, the target table comprising a plurality of text regions, extracts a table structure from the target table image based on positions of the plurality of text regions, the table structure comprising position data of the plurality of text regions distributed in a first direction and position data of the plurality of text regions distributed in a second direction, extracts content data of the target table from the target table image based on the table structure, and outputs the content data of the target table according to the table structure. The table recognition method depends on the position data of the text regions, so it can accurately recognize the target table regardless of whether the target table is horizontal and vertical or twisted, and whether the target table is wired or wireless.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a table recognition method and system. Background Technology

[0002] Spreadsheets are widely used in various fields, such as insurance claims and online office work, due to their ability to effectively store text and data. In insurance claims, insurance calculation is a crucial step, primarily involving extracting detailed amounts from printed expense lists and calculating the required claim amount. Given the high manual cost of insurance calculation, scanning printed spreadsheets into images and then automatically extracting detailed amounts from these images has become the mainstream approach. This automated method first requires recognizing and extracting the table data from the image to generate a spreadsheet.

[0003] Traditional table recognition methods rely on properly aligned table images; that is, the tables in the image must be perfectly straight for accurate recognition. However, in many cases, tables in images are not straight but distorted, such as when the paper used to print the table is curved, wrinkled, or uneven. In such cases, accurate table recognition becomes impossible. Therefore, there is a need for a table recognition method or system that can accurately identify tables. Summary of the Invention

[0004] This specification provides a table recognition method and system that can accurately identify distorted tables in table images.

[0005] In a first aspect, this specification provides a table recognition method, comprising: acquiring a target table image containing a target table, the target table including multiple text regions arranged along the row and / or column directions of the target table, each text region including at least one character; extracting a table structure of the target table from the target table image based on the positions of the multiple text regions, the table structure including position data of the multiple text regions distributed in a first direction and position data of the multiple text regions distributed in a second direction, the first direction being one of the row direction and the column direction, and the second direction being the other of the row direction and the column direction; and extracting content data of the target table from the target table image based on the table structure, and outputting the content data of the target table according to the table structure.

[0006] In some embodiments, obtaining the target table image containing the target table includes: obtaining a to-be-processed image, the to-be-processed image being a scanned image of the target table printed on a medium, the target table image in the to-be-processed image including a distorted and tilted target table; performing tilt correction on the to-be-processed image to correct the tilt of the target table, obtaining a corrected image; and performing table detection on the corrected image to extract the table region in the corrected image, thereby obtaining the target table image containing the target table.

[0007] In some embodiments, extracting the table structure of the target table based on the positions of the plurality of text regions includes: obtaining a two-dimensional first feature map of the target table image, the first feature map including a plurality of pixels; performing a two-dimensional scan of the first feature map in units of pixels to obtain position data of the plurality of text regions distributed in a first direction and position data of the plurality of text regions distributed in a second direction; and combining the position data of the plurality of text regions distributed in the first direction and the position data of the plurality of text regions distributed in the second direction to extract the table structure of the target table.

[0008] In some embodiments, the step of performing a two-dimensional scan of the first feature map in pixels to obtain position data of the plurality of text regions distributed in the first direction and in the second direction includes: obtaining a first projection map of the first feature map in the first direction; using the first projection map as a reference for determining text regions along the first direction, performing a two-dimensional scan of the first feature map in pixels to obtain position data of the plurality of text regions distributed in the first direction; and obtaining a second projection map of the first feature map in the second direction; using the second projection map as a reference for determining text regions along the second direction, performing a two-dimensional scan of the first feature map in pixels to obtain position data of the plurality of text regions distributed in the second direction.

[0009] In some embodiments, obtaining a first projection map of the first feature map in a first direction includes: performing pooling processing on the first feature map in the first direction to obtain the first projection map; obtaining a second projection map of the first feature map in a second direction includes: performing pooling processing on the first feature map in the second direction to obtain the second projection map.

[0010] In some embodiments, the step of using the first projection map as a reference for determining text regions along the first direction, and performing a two-dimensional scan of the first feature map in pixels to obtain the positional data of the distribution of the plurality of text regions in the first direction includes: inputting the first projection map and the first feature map into an attention-memory joint network model to calculate the first feature map pixel by pixel to obtain the positional data of the distribution of the plurality of text regions in the first direction; the step of using the second projection map as a reference for determining text regions along the second direction, and performing a two-dimensional scan of the first feature map in pixels to obtain the positional data of the distribution of the plurality of text regions in the second direction includes: inputting the second projection map and the first feature map into an attention and memory joint model to calculate the first feature map pixel by pixel to obtain the positional data of the distribution of the plurality of text regions in the second direction.

[0011] In some embodiments, the step of performing a two-dimensional scan of the first feature map in pixels to obtain the positional data of the plurality of text regions distributed in the first direction and the positional data of the plurality of text regions distributed in the second direction includes: inputting the first feature map into an attention-memory joint network model trained for the distribution in the first direction to calculate a first target feature vector pixel by pixel on the first feature map, and obtaining the positional data of the plurality of text regions distributed in the first direction based on the first target feature vector; and inputting the first feature map into an attention-memory joint network model trained for the distribution in the second direction to calculate a second target feature vector pixel by pixel on the first feature map, and obtaining the positional data of the plurality of text regions distributed in the second direction based on the second target feature vector.

[0012] In some embodiments, the attention-memory joint network model includes an attention network model and a recurrent memory network model, wherein in the current loop: the recurrent memory network is configured to output the target feature vector of the first feature map and use the target feature vector as the input for its next loop; the attention network model is configured to: receive and, based on the first feature map and the target feature vector output by the recurrent memory network model in the previous loop, determine and output multiple similarity weights corresponding one-to-one with the plurality of pixels, each similarity weight describing the degree of similarity between each pixel in the first feature map and the target feature vector output in the previous loop, and determine and input the target enhancement vector of the first feature map into the recurrent memory network model.

[0013] In some embodiments, the initial input value of the attention-memory joint network model trained for the first directional distribution is a first projection map, which is a projection map of the first feature map in the first direction; and the initial input value of the attention-memory joint network model trained for the second directional distribution is a second projection map, which is a projection map of the first feature map in the second direction.

[0014] In some embodiments, obtaining the positional data of the plurality of text regions distributed in the first direction includes: performing pixel transformation and pixel adjustment processing on the target feature vector of the current cycle to obtain a transformed feature map of the current cycle; performing weighting processing on the first feature map according to the plurality of similarity weights to obtain a weighted feature map; and performing fusion processing on the transformed feature map and the weighted feature map to obtain a second feature map of the current cycle, wherein the second feature map includes the positional data of the plurality of text regions distributed in the first direction.

[0015] In some embodiments, extracting the content data of the target table from the target table image includes: performing character recognition on each of the multiple text regions included in the target table to obtain the text in each text region; obtaining the row header and list header corresponding to each text region; and determining the text in each text region, the row header and list header corresponding to each text region as the content data in the target table.

[0016] Secondly, this specification also provides a table recognition system, including at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for table recognition; the at least one processor is communicatively connected to the at least one storage medium, wherein when the table recognition system is running, the at least one processor reads the at least one instruction set and implements the table recognition method described in the first aspect of this specification.

[0017] As can be seen from the above technical solutions, the table recognition method and system provided in this specification acquire a target table image containing a target table, wherein the target table includes multiple text regions. Based on the positions of the multiple text regions, the table structure is extracted from the target table image. The table structure includes position data of the multiple text regions distributed in a first direction and position data of their distribution in a second direction. Based on the table structure, the content data of the target table is extracted from the target table image, and the content data of the target table is output according to the table structure. The table recognition method of this application depends on the position data of the text regions; therefore, it can accurately recognize the target table regardless of whether it is horizontally or vertically aligned or distorted, and is not limited to wired or wireless tables.

[0018] Other functions of the form recognition methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the form recognition methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The diagram illustrates an application environment of a form recognition system according to some embodiments of this specification;

[0021] Figure 2 A hardware structure diagram of a computing device provided according to some embodiments of this specification is shown;

[0022] Figure 3 An architecture diagram of a table recognition method provided according to some embodiments of this specification is shown;

[0023] Figure 4 A schematic diagram of an image to be processed is shown according to some embodiments of this specification;

[0024] Figure 5 A schematic diagram of an image to be processed is shown according to some embodiments of this specification;

[0025] Figure 6 A schematic diagram of an image to be processed is shown according to some embodiments of this specification;

[0026] Figure 7 A schematic diagram of an image to be processed is shown according to some embodiments of this specification;

[0027] Figure 8 A flowchart of a table recognition method provided according to some embodiments of this specification is shown;

[0028] Figure 9 A flowchart of another table recognition method provided according to some embodiments of this specification is shown;

[0029] Figure 10 A flowchart of another table recognition method provided according to some embodiments of this specification is shown;

[0030] Figure 11 This diagram illustrates an example of inputting a first projection map and a first feature map into an attention-memory joint network model to obtain first location data, according to some embodiments of this specification.

[0031] Figure 12 This diagram illustrates an example of acquiring first position data based on an attention-memory joint network model trained for a first directional distribution, according to some embodiments of this specification; and

[0032] Figure 13 A schematic diagram of table recognition based on an attention-memory joint network model is shown according to some embodiments of this specification. Detailed Implementation

[0033] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0034] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0035] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0036] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0037] Figure 1 This diagram illustrates an application environment for a table recognition system 001 provided according to an embodiment of this specification. The table recognition method provided in this specification can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the table recognition system 001 may include a client 100, a server 200, a computing device 300, a network 400, and a database 500.

[0038] like Figure 1As shown, user 110 is the user of client 100. Client 100 is generally the connection device through which user 110 establishes communication with server 200. User 110 can use client 100 to interact with server 200 via network 400 to receive or send messages, etc. Client 100 and server 200 are connected. In some embodiments, server 200 can simultaneously communicate with multiple clients 100. In some embodiments, client 100 may have one or more applications (APPs) installed. The APP provides user 110 with the ability and interface to interact with the outside world via network 400. The APP includes, but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, client 100 may include mobile devices, tablets, laptops, scanners, built-in devices in motor vehicles or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices or similar devices, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or any combination thereof. In some embodiments, the smart mobile device may include a smartphone, a personal digital assistant, a gaming device, a navigation device, or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include a virtual reality headset, virtual reality glasses, a virtual reality patch, an augmented reality headset, augmented reality glasses, an augmented reality patch, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, a head-mounted display, Gear VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle TV, etc. In some embodiments, the client 100 may be a device with positioning technology for locating the position of the client 100.

[0039] Server 200 may be a server that provides various services, such as a backend server that supports the pages displayed on client 100. In some embodiments, server 200 may include hardware devices with data processing capabilities and the necessary programs to drive the hardware devices. In some embodiments, server 200 may store real-time data generated during the operation of the APP, i.e., an online message queue.

[0040] The computing device 300 may store data or instructions for performing the table recognition method described in this specification, and may execute or be used to execute said data or instructions. In some embodiments, the computing device 300 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device. The computing device 300 may be communicatively connected to the server 200. In some embodiments, the computing device 300 may communicate with the server 200 via a network 400. In some embodiments, the computing device 300 may communicate with the server 200 via a wired connection. In some embodiments, the computing device 300 may be integrated into the server 200, as part of the server 200. In some embodiments, the computing device 300 may be a device independent of the server 200. In some embodiments, the computing device 300 may be a distributed computing device, including multiple distributed computing nodes. It should be noted that the table recognition method provided in this specification is generally executed by the computing device 300.

[0041] It should be understood that Figure 1 The number of clients 100, servers 200, computing devices 300, networks 400, and databases 500 shown is merely illustrative. Depending on implementation needs, any number of clients 100, servers 200, computing devices 300, networks 400, and databases 500 can be included.

[0042] Network 400 serves as a medium for providing a communication connection between client 100 and server 200. In some embodiments, network 400 can also be used as a medium for providing a communication connection between server 200, computing device 300, and database 500. Network 400 can facilitate the exchange of information or data. Figure 1As shown, client 100, server 200, computing device 300, and database 500 can connect to network 400 and transmit information or data to each other through network 400. For example, server 200 can obtain information from client 100 through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 can include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. In some embodiments, network 400 can include one or more network access points. For example, network 400 can include wired or wireless network access points, such as base stations or Internet switching points, through which one or more components of client 100, server 200, computing device 300, and database 500 can connect to network 400 to exchange data or information.

[0043] Database 500 may store data or instructions. In some embodiments, database 500 may store data obtained from server 200 or client 100. In some embodiments, database 500 may store data or instructions that server 200 can execute or use to execute the table recognition method described in this specification. In some embodiments, database 500 may store offline data generated during the runtime of the APP. Computing device 300, server 200, and client 100 may have permission to access database 500, and computing device 300, server 200, and client 100 may access data or instructions stored in database 500 via network 400. In some embodiments, database 500 may be directly connected to computing device 300, server 200, and client 100. In some embodiments, database 500 may be part of server 200. In some embodiments, database 500 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or similar content, or any combination thereof. Exemplary mass storage may include non-transitory storage media such as disks, optical disks, and solid-state drives. Removable storage may include flash drives, floppy disks, optical discs, memory cards, zip disks, magnetic tapes, etc. Typical volatile read / write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), dual date rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero-capacitance RAM (Z-RAM), etc. ROM may include mask ROM (MROM), programmable ROM (PROM), virtual programmable ROM (PEROM), electronically programmable ROM (EEPROM), optical disc (CD-ROM), and digital multifunction disk ROM, etc. In some embodiments, database 500 may be implemented on a cloud platform. By way of example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, etc., or similar forms, or any combination of the above forms.

[0044] like Figure 1 As shown, user 110 sends a table recognition request through the APP on client 100. The table recognition request is transmitted to server 200 through network 400. Server 200 transmits the table recognition request to computing device 300 through a communication connection with computing device 300. Computing device 300 executes the instructions of the table recognition method stored in computing device 300, and recognizes and outputs the target table in the target table image based on the table recognition request.

[0045] Figure 2A hardware structure diagram of a computing device 300 provided according to an embodiment of this specification is shown. The computing device 300 can execute the table recognition method described in this specification. The table recognition method will be described in detail below. The computing device 300 may include at least one processor 320 and at least one storage medium 330. In some embodiments, the computing device 300 may further include a communication module 350 and an internal communication bus 310.

[0046] The internal communication bus 310 can connect different system components, including storage medium 330, processor 320 and communication module 350.

[0047] Storage medium 330 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 332, a read-only storage medium (ROM) 334, or a random access storage medium (RAM) 336. Storage medium 330 may store at least one instruction set. The instruction set may be a collection of one or more instructions. For example, the instruction set may be multiple lines of computer program code, which, when retrieved and executed by processor 320, may instruct processor 320 to execute programs, routines, objects, components, data structures, procedures, modules, etc., of the table recognition method provided in this specification.

[0048] At least one processor 320 may be communicatively connected to at least one storage medium 330. The at least one processor 320 is used to read / recall and execute the at least one instruction set described above. When the computing device 300 is running, the at least one processor 320 reads the at least one instruction set and executes the table recognition method provided in this specification according to the instructions of the at least one instruction set. The processor 320 may execute all the steps included in the table recognition method. The processor 320 may be in the form of one or more processors. In some embodiments, the processor 320 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof. For illustrative purposes only, only one processor 320 is described in this specification for the computing device 300. However, it should be noted that the computing device 300 in this specification may also include multiple processors 320. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described in this specification, or they may be executed jointly by multiple processors. For example, if the processor 320 of the computing device 300 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 320 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0049] Of course, the module that performs the table recognition method provided in this specification can also be designed as a dedicated circuit, hardware module and / or device, independently communicating with the computing device 300 or existing as an internal component within the computing device 300. Figure 2 (Not marked).

[0050] The communication module 350 can be connected to the processor 320 for data communication between the computing device 300 and external devices, such as the server 200 and the database 500. The communication module 350 may include at least one of a wired communication module and a wireless communication module.

[0051] Figure 3 A flowchart 3000 illustrating a table recognition method flow according to embodiments of this specification is shown. Figure 3As shown in the diagram, the architecture of the table recognition process 3000 includes an input module 3100, a preprocessing module 3200, a structure recognition module 3300, a character recognition module 3400, a structure cognition module 3500, and an output module 3600. Each module can be one or more instruction sets stored in the storage medium 330 that execute corresponding instructions, and is executed by the processor 320 in the computing device 300. Each module can also be a part of the circuitry, hardware device, or module within the computing device 300. For example, the input module 3100 can be a hardware device / module in the computing device 300 that receives the image to be processed, and the preprocessing module 3200 can be a hardware device / module in the computing device 300 that preprocesses the image to be processed, and so on.

[0052] The input module 3100 can receive the image to be processed and output the image to be processed to the preprocessing module 3200.

[0053] The image to be processed is a scanned image obtained by scanning a target table printed on a medium. The scanning can be done by scanning with a scanner or by taking a picture with a mobile phone. When scanning a target table printed on a medium, the area covered by the scanning window of the scanning device (scanner or mobile phone) may include not only the area where the target table is located but also the surrounding background area. Therefore, to distinguish between the area where the target table is located and the background area in the image to be processed, the image area where the target table is located can be referred to as the target table image. "Printed on a medium" can include printing on paper or on other media, such as printing on film or displaying on a screen, etc., any medium that can visually present the table content. The embodiments in this specification do not limit this.

[0054] Taking paper as an example, the target table printed on paper is usually rectangular, such as... Figure 4 The diagram illustrates a schematic representation of an image to be processed, comprising a rectangular target table printed on paper. The target table includes several text areas, each of which can be filled with one or more characters. The filled characters can be from any language, such as Chinese, English, Russian, etc., or Arabic numerals or any symbols. This specification does not specifically limit the type of filled characters in the embodiments. Figure 4 In the text, the area occupied by the words "operation type" is a text area, the area occupied by the Arabic numeral "1" is a text area, and the area occupied by the symbol "—" is also a text area.

[0055] The target table is divided into row and column directions. Several text regions are arranged along the row and / or column directions. In a wired target table, the text regions are separated along the row and / or column directions by dividing lines, which can be dotted or solid. In a wireless target table, the text regions are naturally displayed along the row and / or column directions through blank areas. The arrangement of these text regions along the row and / or column directions constitutes the table structure of the target table. In other words, the target table can include a table structure and content arranged according to that structure. The content can be displayed visually on a screen, and the content data is stored in a storage medium as data. Therefore, the image to be processed includes a target table image, which displays the table structure and content of the target table. During table recognition, it is necessary to extract the table structure and content data of the target table.

[0056] The target table contains data including the text within each text area, the row header for each text area, and the column header. The row header includes the text from the first text area when multiple text areas are arranged in a row; that is, the row header includes the text from the first column of text areas. Figure 4 The row headers in the table include "1", "2", ..., "17". For each text region, its corresponding row header is the text in the first text region of the line containing that text region. For example, in... Figure 4 In the target table, the row header for each text area in the second row is the text "1". Similarly, the list header includes the text from the first text area when multiple text areas are arranged along the column direction; that is, the list header includes the text from the first row of text areas. Figure 4 The list header includes "Operation Type," "Parameter Information," "Input Data Dimension," and "Output Data Dimension." For each text region, its corresponding list header is the text in the first text region of the column containing that text region. For example, in... Figure 4 In the target table, the list header for each text area in the second column is the text "Operation Type".

[0057] It should be noted that the row and column directions of the target table change as the target table image is positioned within the image to be processed. When the target table image is relatively upright within the image to be processed, i.e., when the target table is horizontally and vertically aligned, all rows containing the text area are horizontal, and all columns containing the text area are vertical. In other words, within the image to be processed, the row direction of the target table is consistent with the row direction of the pixels in the image to be processed, and the column direction of the target table is consistent with the column direction of the pixels in the image to be processed.

[0058] However, sometimes the target table in the image to be processed may be tilted. For example, if the paper with the target table printed on it was not straightened when it was scanned and converted into an electronic image, then the target table will be tilted in the image to be processed. Figure 5 As shown, Figure 5 This diagram illustrates a scenario where the target table in the image to be processed is tilted. In this case, the row and column directions of the target table have non-zero angles with the row and column directions of the pixel arrangement in the image to be processed, respectively. Sometimes, the target table in the image to be processed may also be distorted. For example, if the paper on which the target table is printed is wrinkled, folded, or warped during scanning (e.g., when a book is opened and a page is photographed, the page is warped), then the target table in the image will appear distorted, with its row and column directions not being straight lines but curves or broken lines. Figure 6 As shown, Figure 6 This diagram illustrates a scenario where the target table in the image to be processed is distorted. For example, if the scanning window of the scanning device is not parallel to the paper plane, the target table reflected in the image will also be distorted. If the paper with the target table printed on it is straight when scanned, but the scanning window plane is not parallel to the paper plane, the target table reflected in the image will appear trapezoidal. Sometimes, the target table in the image to be processed may be both tilted and distorted. For example, if the paper with the target table printed on it is not straight and is folded when scanned, the target table reflected in the image will be both tilted and distorted. Figure 7 As shown, Figure 7 A schematic diagram is shown of a target table in an image to be processed that is both tilted and distorted. Furthermore, the target table in the image to be processed can also be dense or sparse; the embodiments in this specification are not limited to this.

[0059] The preprocessing module 3200 can preprocess the image to be processed to obtain a target table image containing the target table, and then output the target table image to the structure recognition module 3300. Specifically, the target table in the image to be processed may be tilted or upright. When it is necessary to identify and process the tilt of the target table, the preprocessing module 3200 may include an orientation model 3201; when it is not necessary to identify and process the tilt of the target table, the preprocessing module 3200 may not include an orientation model 3201. When it is necessary to extract the table region in the image to be processed, the preprocessing module 3200 may include a table detection model 3202; when it is not necessary to extract the table region in the image to be processed, the preprocessing module 3200 may not include a table detection model 3202. The orientation model 3201 can perform tilt correction on the image to be processed to correct the tilt of the target table in the image to be processed, obtain a corrected image, and output the corrected image to the table detection model 3202. The table detection model 3202 can perform table detection on the corrected image, extract the table region in the corrected image, obtain a target table image containing the target table, and then output the target table image to the structure recognition module 3300.

[0060] The structure recognition module 3300 can extract the table structure of the target table image based on the positions of multiple text regions, and output the table structure to the text recognition module 3400. Specifically, the structure recognition module 3300 may include a column model 3301, a row model 3302, and a merging model 3303. The column model 3301 can acquire a two-dimensional first feature map of the target table image, perform a two-dimensional scan of the first feature map in pixels, acquire the position data of the multiple text regions in the column direction, and output the position data of the multiple text regions in the column direction to the merging model 3303. The row model 3302 can acquire a two-dimensional first feature map of the target table image, perform a two-dimensional scan of the first feature map in pixels, acquire the position data of the multiple text regions in the row direction, and output the position data of the multiple text regions in the row direction to the merging model 3303. The merging model 3303 can combine the positional data of the multiple text regions distributed in the column direction and the positional data of their distribution in the row direction to extract the table structure of the target table, and output the table structure to the text recognition module 3400. The method by which the structure recognition module 3300 extracts the table structure of the target table from the target table image based on the positions of multiple text regions will be described in detail later.

[0061] The text recognition module 3400 and the structure recognition module 3500 can extract the content data of the target table from the target table image based on the table structure. The content data includes the text in each text area, the row header and the list header corresponding to each text area. The structure recognition module 3500 can output the content data of the target table through the output module 3600 according to the table structure.

[0062] The text recognition module 3400 can perform text recognition on each of the multiple text regions included in the target table, obtain the text within each text region, and output the text within each text region to the structure recognition module 3500. The text recognition module 3400 may include only a text recognition model 3401, or it may include both a text recognition model 3401 and a post-processing model 3402. When the text recognition module 3400 includes only a text recognition model 3401, the text recognition model 3401 can recognize the text within each text region of the target table image, obtain the text within each text region, and output the text within each text region to the structure recognition module 3500. When the text recognition module 3400 includes both a text recognition model 3401 and a post-processing model 3402, the text recognition model 3401 can recognize the text within each text region of the target table image, obtain the initial text within each text region, and then output the initial text to the post-processing model 3402. The post-processing model 3402 can post-process the initial text to obtain the text within each text region, and output the text within each text region to the structure recognition module 3500. The post-processing includes further processing of text that the text recognition model 3401 cannot recognize, such as confirming the text based on its context.

[0063] The structure recognition module 3500 may include a row header model 3501 and a structure recognition model 3502. The row header model 3501 can obtain the row header and list header corresponding to each text region; the structure recognition model 3502 can recognize (determine) the text in each text region, the row header corresponding to each text region, and the list header corresponding to each text region as the content data in the target table, and output the content data of the target table through the output module 3600 according to the table structure.

[0064] The output module 3600 can output the content data of the target table according to the table structure. For example, the output module 3600 can output the content data in the form of a table structured JSON, where JSON is a data organization method that describes data in a structured form.

[0065] Figure 8 A flowchart of a table recognition method P100 according to an embodiment of this specification is shown. As previously described, computing device 300 can execute the table recognition method P100 described in this specification. Specifically, processor 320 can read an instruction set stored in its local storage medium and then execute the table recognition method P100 described in this specification according to the provisions of the instruction set. Figure 8 As shown, method P100 may include:

[0066] S110: Obtain the target table image containing the target table.

[0067] like Figure 9 As shown, step S110 may further include steps S111-S113:

[0068] S111: Obtain the image to be processed, which is a scanned image of a target table printed on a medium, and the target table image in the image to be processed includes a distorted and tilted target table.

[0069] Client 100 can acquire an image to be processed and send the image to be processed to computing device 300 through server 200, thereby obtaining the image to be processed.

[0070] As mentioned earlier, the target table in the image to be processed may be tilted, distorted, or both tilted and distorted. If the target table in the image to be processed is not tilted, the tilt correction process in S112 can be skipped, and the table detection process in S113 can be performed directly.

[0071] S112: The image to be processed is tilt-corrected to correct the tilt of the target table, resulting in a corrected image.

[0072] To improve the accuracy of table recognition, the processor 320 needs to perform tilt correction on the image to be processed to correct the tilt of the target table. The processor 320 can use a classification model based on CNN (Convolutional Neural Networks) to perform tilt correction on the image to be processed. Of course, other tilt correction methods can also be used to correct the tilt of the image to be processed, such as methods based on Hough transform, projection-based methods, and k-nearest neighbor clustering. The embodiments in this specification are not limited to these methods.

[0073] S113: Perform table detection on the corrected image, extract the table region in the corrected image, and obtain the target table image containing the target table.

[0074] The processor 320 extracts the table region from the corrected image to obtain a target table image containing the target table. That is, it removes other regions from the corrected image except for the table region. In this way, by performing table detection on the corrected image and separating the table region from other regions, only the target table image containing the target table is retained, which improves the efficiency of table recognition.

[0075] The processor 320 can use the RFCOS model to perform table detection on the corrected image. Of course, other methods can also be used to perform table detection on the corrected image. The embodiments in this specification do not limit this.

[0076] After obtaining the target table image, method P100 may further include:

[0077] S120: Extract the table structure of the target table from the target table image based on the positions of multiple text regions.

[0078] For ease of description, this specification adopts the following definition: the row direction and column direction of the target table are divided into a first direction and a second direction. That is, if the first direction is the row direction, then the second direction is the column direction, and vice versa. Accordingly, the table structure includes position data of the plurality of text regions distributed in the first direction and position data of the plurality of text regions distributed in the second direction. For ease of description, in the subsequent detailed explanation of the steps, the position data of the plurality of text regions distributed in the first direction will be referred to as the first position data, and the position data of the plurality of text regions distributed in the second direction will be referred to as the second position data.

[0079] The first position data can be the sequential values ​​of the plurality of text regions in a first direction. Taking the first direction as a column direction as an example, the first position data can include column 1, column 2, column 3, and column 4. The second position data can be the sequential values ​​of the plurality of text regions in a second direction. Taking the second direction as a row direction as an example, the second position data can be row 1, row 2, and row 3. The first position data can also be the pixel coordinate value of each pixel in the pixels covered by each text region in the first direction, or it can be any pixel selected from the covered pixels, and the pixel coordinate value of the arbitrary pixel in the first direction used as the first position data. The second position data can also be the pixel coordinate value of each pixel in the pixels covered by each text region in the second direction, or it can be any pixel selected from the covered pixels, and the pixel coordinate value of the arbitrary pixel in the second direction used as the second position data. The embodiments of this specification do not limit the specific content of the first and second position data.

[0080] Specifically, such as Figure 10 As shown, step S120 may include steps S121-S123:

[0081] S121: Obtain a two-dimensional first feature map of the target table image, wherein the first feature map includes multiple pixels.

[0082] The processor 320 can use a CNN model to downsample and extract features from the target table image to obtain a two-dimensional first feature map of the target table image. For example, an HRNet (High-Resolution Network) model can be used for the downsampling and feature extraction. When obtaining the first feature map, the size of the target table image can be adjusted first, and then the HRNet model can be used to downsample and extract features from the adjusted target table image to obtain the first feature map. The size of the adjusted target table image can be, for example, 320*320, and the size of the first feature map can be, for example, 80*80. The embodiments in this specification do not limit the size of the adjusted target table image or the size of the first feature map. Of course, other methods can also be used to extract features from the target table image to obtain the first feature map, such as using a VGG-Net (Visual Geometry Group Network) model, a GoogLeNet model, etc. The embodiments in this specification do not limit the method for obtaining the first feature map.

[0083] S122: Perform a two-dimensional scan of the first feature map in pixels to obtain the position data of the multiple text regions distributed in the first direction and the position data of the multiple text regions distributed in the second direction.

[0084] As mentioned earlier, the table structure includes the first position data and the second position data. In this step, the processor 320 needs to determine the timing of acquiring the first position data and the timing of acquiring the second position data, that is, it needs to determine when the first position data is acquired and when the second position data is acquired. There are two possible ways to determine the timing of acquiring the first position data and the timing of acquiring the second position data:

[0085] One possible implementation: First, the processor 320 acquires a first projection map of the first feature map in the first direction; and acquires a second projection map of the first feature map in the second direction. For example, the processor 320 can perform pooling on the first feature map in the first direction to acquire the first projection map. Similarly, the processor 320 can perform pooling on the first feature map in the second direction to acquire the second projection map. The pooling process can be average pooling or max pooling. For a two-dimensional feature map, the pooled projection map is a one-dimensional array, and its value reflects the accumulation of pixel values ​​of the feature map along the corresponding pooling direction, that is, the number of characters corresponding to the feature map at a certain position along the pooling direction. For example, if pooling is performed along the column direction, the result is the projection of all text feature maps along the column direction. This projection along the column direction can provide a reference for the processor 320 to subsequently recognize the text region along the column direction.

[0086] It should be noted that after pooling the first feature map in the first direction, the pooled image can be input into a temporal model for mutual perception processing, so that each text region in the pooled image contains information about other text regions, thereby outputting a first projection map of the text regions mutually perceiving each other in the second direction. For example, the temporal model can be a BLSTM (Bidirectional Long Short-Term Memory) network model or a ConvLSTM (Convolutional Long Short-Term Memory) network model, etc.

[0087] For example, assuming the first direction is the column direction, and the pooled image has a height of 1 and a width of W, and includes four text regions arranged horizontally, then when the pooled image is input into a BLSTM temporal model, these four text regions are mutually perceptive in the horizontal direction, i.e., mutually perceptive in the row direction. Each text region includes information from the other three text regions, thus outputting a first projection map of the text regions mutually perceptive in the second direction. Similarly, after pooling the first feature map in the second direction, the pooled image can also be input into a temporal model for mutual perceptive processing, so that each text region in the pooled image contains information from the other text regions, thus outputting a second projection map of the text regions mutually perceptive in the first direction.

[0088] It should also be noted that the first projection map shows the shadow distribution of the text region in the first feature map along the first direction. Taking the first direction as the column direction as an example, for a target table image containing horizontal and vertical target tables, a column of text regions will be projected onto the same projection area along the column direction, and a column of blank regions will also be projected onto the same projection area along the column direction. The projection value of each pixel in the projection area corresponding to a column of text regions is greater than the projection value of each pixel in the projection area corresponding to a column of blank regions. The same applies when the first direction is the row direction. Therefore, based on the projection map of the target table image containing horizontal and vertical target tables, the number of columns and rows of text regions in the target table can be determined, and the text regions and blank regions can be clearly distinguished.

[0089] Taking the column direction as an example, for a target table image containing a distorted target table, a column of text regions will be projected onto different projection regions along the column direction. These projection regions may partially overlap, or they may not overlap. That is, the projection map of a target table image containing a distorted target table can include overlapping areas of text regions, non-overlapping areas of text regions, and blank areas. Pixels in overlapping areas are more likely to be in the middle part of the text region, while pixels in non-overlapping areas are more likely to be in the boundary part of the text region. The projection value of each pixel in the overlapping area is greater than that in the non-overlapping area, and the projection value of each pixel in the non-overlapping area is greater than that in the blank area. The same principle applies when the first direction is the row direction. Therefore, based on the projection map of a target table image containing a distorted target table, we can obtain the approximate number of columns and rows of the text region, and we can also determine whether each pixel in the projection map is more likely to be in the middle part of the text region, a pixel in the boundary part of the text region, or a pixel in the blank area.

[0090] It can be seen that regardless of whether the target table is horizontally or vertically straight, or distorted, the row and column numbers of the text area, the pixel information of the text area, and the pixel information of the blank area can all be obtained from the projection map of the target table image. Therefore, after obtaining the first feature map and its first projection map in the first direction, the first projection map can be used as a reference for determining the text area along the first direction to obtain the first position data. After obtaining the first feature map and its second projection map in the second direction, the second projection map can be used as a reference for determining the text area along the second direction to obtain the second position data.

[0091] It should be noted that if the first direction is the column direction, the first projection image includes column number information, pixel information of the text areas, and pixel information of the blank areas. Based on the first projection image, it can be determined that the positional data of multiple text areas distributed along the column direction is to be acquired. If the first direction is the row direction, the first projection image includes row number information, pixel information of the text areas, and pixel information of the blank areas. Based on the first projection image, it can be determined that the positional data of multiple text areas distributed along the row direction is to be acquired. Thus, the timing for acquiring the first positional data can be determined based on the first projection image. Similarly, the timing for acquiring the second positional data can be determined based on the second projection image.

[0092] After obtaining the first projection map of the first feature map in the first direction, the processor 320 uses the first projection map as a reference for determining the text region along the first direction, and performs a two-dimensional scan of the first feature map in pixels to obtain the positional data of the distribution of the multiple text regions in the first direction. Specifically, the processor 320 can input the first projection map and the first feature map into an attention-memory joint network model to calculate the first feature map pixel by pixel to obtain the positional data of the distribution of the multiple text regions in the first direction.

[0093] For example, Figure 11 This diagram illustrates an embodiment of the present specification where a first projection map and a first feature map are input into an attention-memory joint network model to obtain first location data. (See attached diagram.) Figure 11As shown, the target table image includes a distorted target table; the target table comprises four columns of text regions, represented by three black boxes for the first column, four boxes filled with left diagonal lines for the second column, three boxes filled with right diagonal lines for the third column, and three boxes filled with black dots for the fourth column. The three diagonal lines in the target table image are not real; they are only used to illustrate the distortion of the target table. The target table image is processed by a CNN to obtain a first feature map with a width of W pixels and a height of H pixels. The first feature map is then pooled to obtain an image with a width of W pixels and a height of 1 pixel. The pooled image is then processed by a BLSTM to output a first projection map. The first projection map and the first feature map are input into an attention-memory joint network model Z to calculate the first feature map pixel by pixel, thereby obtaining the first position data.

[0094] Similarly, after obtaining the second projection map of the first feature map in the second direction, the processor 320 can use the second projection map as a reference to determine the text region along the second direction, and perform a two-dimensional scan of the first feature map in pixels to obtain the positional data of the multiple text regions distributed in the second direction. Specifically, the processor 320 can input the second projection map and the first feature map into an attention and memory joint model to calculate the first feature map pixel by pixel to obtain the positional data of the multiple text regions distributed in the second direction. The process of obtaining the second positional data is similar to that of obtaining the first positional data, and will not be described in detail here.

[0095] It should be noted that the embodiments in this specification perform two-dimensional scanning of the first feature map in units of pixels. Two-dimensional scanning refers to scanning the pixels of the image by row and column. The first feature map is calculated pixel by pixel through a joint attention and memory model. Therefore, tables can be identified with relatively fine granularity. Even if the target table image is a relatively dense image, the target table can be accurately identified.

[0096] A second possible implementation involves inputting the first feature map into an attention-memory joint network model trained for the first directional distribution to calculate a first target feature vector pixel by pixel, and obtaining positional data of the multiple text regions distributed in the first directional direction based on the first target feature vector; and inputting the first feature map into an attention-memory joint network model trained for the second directional distribution to calculate a second target feature vector pixel by pixel, and obtaining positional data of the multiple text regions distributed in the second directional direction based on the second target feature vector.

[0097] The first target feature vector contains information about the first position data, and the second target feature vector contains information about the second position data.

[0098] In the second possible implementation, first position data and second position data are obtained based on a specially trained attention-memory joint network model. If the attention-memory joint network model is trained for a first directional distribution, the first position data can be obtained based on the trained attention-memory joint network model. If the attention-memory joint network model is trained for a second directional distribution, the second position data can be obtained based on the trained attention-memory joint network model.

[0099] For example, Figure 12 A schematic diagram is shown illustrating an embodiment of this specification of obtaining first position data based on an attention-memory joint network model trained for a first directional distribution. For example... Figure 12 As shown, Y is an attention-memory joint network model trained for the first directional distribution. The first feature map is input into Y, and the first target feature vector h is obtained by calculating the first feature map pixel by pixel. t And based on h t Get the first position data.

[0100] In some embodiments, the initial input value of the attention-memory joint network model trained for the first directional distribution is a first projection map, which is a projection map of the first feature map in the first direction; and the initial input value of the attention-memory joint network model trained for the second directional distribution is a second projection map, which is a projection map of the first feature map in the second direction.

[0101] It should be noted that the first projection map used as the initial input value can be the first projection map in the first possible implementation, and the second projection map used as the initial input value can be the second projection map in the first possible implementation. The embodiments of this specification will not elaborate further on the first and second projection maps. Since the first and second projection maps themselves contain information about the target table image, using the first and second projection maps as initial input values ​​can improve the table recognition effect. Of course, the initial input values ​​for the attention-memory joint network model trained for the first directional distribution and the attention-memory joint network model trained for the second directional distribution can also be other values, such as 0. The embodiments of this specification do not limit this.

[0102] It should also be noted that if the attention-memory joint network model is trained for the first directional distribution, then it can be determined that the data to be acquired is at the first position. If the attention-memory joint network model is trained for the second directional distribution, then it can be determined that the data to be acquired is at the second position. Thus, the timing for acquiring the first position data and the timing for acquiring the second position data can be determined based on the specially trained attention-memory joint network model.

[0103] The processor 320 may use the first possible implementation to determine the timing of acquiring the first position data and the timing of acquiring the second position data, that is, to determine the timing based on the first projection map and the second projection map; it may also use the second possible implementation to determine the timing of acquiring the first position data and the timing of acquiring the second position data, that is, to determine the timing based on a specially trained attention-memory joint network model; or it may use both the first possible implementation and the second possible implementation simultaneously. The embodiments in this specification do not limit this.

[0104] It should be noted that there are dividing lines between multiple text regions in the target table with lines. These dividing lines are usually located between two adjacent text regions. Therefore, this embodiment can also use these dividing lines to assist in obtaining the positions of the multiple text regions. For example, for a horizontally or vertically aligned target table or a distorted target table, the processor 320 can obtain the projection area of ​​the dividing line and use this projection area to assist in determining the position of the text region. In addition, this embodiment can also use a text recognition device to perform text recognition before obtaining the position of the text region, using the text recognition device to assist in obtaining the position of the text region. However, it should be clarified that this embodiment can independently identify and extract text regions. That is, the above method does not depend on whether there are dividing lines between the multiple text regions when identifying and extracting text regions. The text regions of the target table can be determined regardless of whether the dividing lines exist. At the same time, the above method can also assist in identifying multiple text regions without relying on text recognition at all. That is, the above method can identify text regions without recognizing the text in the target table.

[0105] S123: Combining the position data of the multiple text regions distributed in the first direction and the position data distributed in the second direction, extract the table structure of the target table.

[0106] The processor 320 acquires the first position data and the second position data, that is, it acquires the position data of the multiple text regions distributed in the column direction and the position data of the multiple text regions distributed in the row direction, and thus determines the position data of the multiple text regions in the target table. For example, the position data of a certain text region in the target table is (the nth row and the nth column), and thus the table structure of the target table is extracted.

[0107] As mentioned earlier, the process of recognizing and extracting text regions can be aided by dividing lines and / or a text recognizer. That is, the process of extracting the table structure can refer to the dividing lines or the recognition of text in the target table. However, when recognizing and extracting text regions, the processor 320 can also identify multiple text regions without relying on the dividing lines of the target table, or without recognizing the text in the target table. Instead, it can use an attention-memory joint network model to judge the relevance of each text region in the target table image pixel by pixel to identify multiple text regions. Therefore, in the entire process of extracting the table structure, it is possible to extract the table structure solely based on the position of the text regions, without depending on whether the target table is based on dividing lines or without recognizing the text in the target table.

[0108] like Figure 8 As shown, after extracting the table structure of the target table, the method P100 may further include step S130: based on the table structure, extracting the content data of the target table from the target table image, and outputting the content data of the target table according to the table structure.

[0109] In this step, the processor 320 can extract the content data of each text region according to the first position data and the second position data of each text region. For example, the processor 320 can extract the content data of the text region in the first column and the first row, and extract the content data of the text region in the second column and the first row. The content data includes the text in each text region and the row header and column header corresponding to each text region.

[0110] For example, the processor 320 can perform text recognition on each of the multiple text regions included in the target table to obtain the text in each text region; obtain the row header and list header corresponding to each text region; and determine the text in each text region, the row header and list header corresponding to each text region as the content data in the target table.

[0111] It should be noted that, as mentioned earlier, the row header typically includes the text in the first column of the text area, and the list header typically includes the text in the first row of the text area. The row header corresponding to each text area is the text in the first text area of ​​the row containing that text area, and the list header corresponding to each text area is the text in the first text area of ​​the column containing that text area. Therefore, for any text area other than the row header and list header, the processor 320 can store the corresponding row header and list header of that text area along with the text within that text area in the storage medium as double-key-value pairs. Here, the "double key" includes the corresponding row header and list header of the text area, and the "value" is the text within that text area. That is, the processor 320 can use the corresponding row header and list header to interpret the text within that text area. For example, in... Figure 4 If the row header for "convolution operation" is "1" and the column header is "operation type", then the processor 320 can determine that "convolution operation" is the first operation type.

[0112] Alternatively, OCR (Optical Character Recognition) technology can be used to recognize the text within each text region.

[0113] It should be noted that the processor 320 can obtain the row header and list header corresponding to each text region through different methods. In some embodiments, the processor 320 can directly extract the text from the first text region when the multiple text regions are arranged along the row direction of the target table, as the row header of each text region when the multiple text regions are arranged along the row direction of the target table; the processor 320 can also directly extract the text from the first text region when the multiple text regions are arranged along the column direction of the target table, as the list header of each text region when the multiple text regions are arranged along the column direction of the target table. In some embodiments, the processor 320 can input the table structure of the target table and the text within the multiple text regions into a graph network model, and obtain the row header and list header corresponding to each text region output by the graph network model.

[0114] In some embodiments, the processor 320 can output the content data of the target table to a target medium according to the table structure. The target medium may include a storage medium or a display screen. For example, the processor 320 can store the table structure and content data of the target table in the storage medium in matrix form or JSON form; or, it can display the table structure and content data of the target table on the display screen as a spreadsheet image. Of course, the target medium can also be other devices, such as servers, etc., and the embodiments in this specification are not limited thereto.

[0115] In step S122, both the first possible implementation and the second possible implementation involve an attention-memory joint network model. The first position data and the second position data are obtained based on the attention-memory joint network model. The following describes in detail the method of table recognition based on the attention-memory joint network model.

[0116] According to some embodiments of this application, the attention-memory joint network model may include an attention network model connected to a recurrent memory network model, and operate in a cyclic iterative and feedback manner. In the current loop: the recurrent memory network is configured to output the target feature vector of the first feature map and use the target feature vector as input for its next loop; the attention network model is configured to: receive and, based on the first feature map and the target feature vector output by the recurrent memory network model in the previous loop, determine and output multiple similarity weights corresponding one-to-one with the plurality of pixels, each similarity weight describing the degree of similarity between each pixel in the first feature map and the target feature vector output in the previous loop; and determine and input the target enhancement vector of the first feature map into the recurrent memory network model.

[0117] It should be noted that if step S122 adopts the second possible implementation method, the target feature vector output by the recurrent memory network B can be a first target feature vector obtained based on the attention-memory joint network model trained for the first directional distribution, or a second target feature vector obtained based on the attention-memory joint network model trained for the second directional distribution.

[0118] In some embodiments, the attention network model A employs a two-dimensional attention (2D Attention) mechanism. The role of this mechanism is to assign different weights to different regions in the image, identify regions of interest (ROIs), strengthen these ROIs, suppress uninterested regions, and output an attention probability map or probability matrix. ROIs are highlighted in the probability map, and their corresponding probability values ​​are higher in the probability matrix. Attention mechanisms include Hard Attention and Soft Attention. Hard Attention can only strengthen one region at a time, while Soft Attention can detect and strengthen multiple regions at a time. The two-dimensional attention mechanism in this application uses Soft Attention. Additionally, the recurrent memory network model B can be an LSTM (Long Short-Term Memory) model.

[0119] For example, Figure 13A schematic diagram illustrating table recognition based on an attention-memory joint network model according to an embodiment of this specification is shown. Figure 13 As shown, the attention-memory joint network model includes attention network model A and recurrent memory network model B. Attention network model A uses a 2D attention model, and recurrent memory network model B uses an LSTM model, which includes multiple LSTMs. Figure 13 The following example uses three LSTMs. Each LSTM corresponds to a 2D Attention model; that is, the three LSTMs actually correspond to three 2D Attention models. Figure 13 The diagram only schematically illustrates a 2D Attention model in the current loop. Additionally, Figure 13 Continued Figure 11 The schematic diagram of the process of inputting the first projection image into the attention-memory joint network model.

[0120] refer to Figure 13 Taking the second LSTM as the current iteration of the LSTM as an example, in the current iteration (i.e., at the current time t), the LSTM outputs the target feature vector h. t and h t The input is fed into the next LSTM. The 2D Attention model receives the first feature map F and the target feature vector h output by the first LSTM. t-1 Based on F and h t-1 Output α t and target enhancement vector g t α t It consists of multiple similarity weights that correspond one-to-one with the multiple pixels, and g t Input into LSTM.

[0121] It should be noted that the embodiments in this specification exemplarily describe the process of attention network model A and recurrent memory network model B in the current cycle. The process of other cycles of attention network model A and recurrent memory network model B is the same as the process of the current cycle, and the embodiments in this specification will not be described again here.

[0122] In some embodiments, the attention network model A can output α using Equations 1 and 2. ij :

[0123] Formula 1: e ij =tanh(w F F ij +w F h t-1 )

[0124] Among them, F ij For the pixel in the i-th row and j-th column of the first feature map F, the first feature map F includes multiple channels, F ij It is a one-dimensional vector composed of the pixel values ​​of each pixel in multiple channels, e ij Let w be the initial weight corresponding to the pixel in the i-th row and j-th column of the first feature map F, tanh() is the similarity function, and w is the weight corresponding to the pixel in the j-th column of the first feature map F. F For fixed weights, h t-1 The target feature vector output by the recurrent memory network model B in the previous iteration.

[0125] Formula 2:

[0126] Where, α ij Let α be the similarity weight corresponding to the pixel in the i-th row and j-th column of the first feature map F. ij The value range is [0,1], and softmax() is the normalization function. The weights are fixed.

[0127] In some embodiments, the attention network model A can output the target enhancement vector g using Equation 3. t :

[0128] Formula 3: g t =∑ ij α ij F ij ,i=1,L,H;j=1,L,W.

[0129] Among them, g t The target enhancement vector is a one-dimensional vector, ∑ ij α ij F ij , i = 1, L, H; j = 1, L, W. refers to the weight α for each similarity. ij And each pixel F in the first feature map F ij The features are multiplied, and the results of multiple multiplications are added together. The width of the first feature map F is H, and the height is W.

[0130] In some embodiments, the recurrent memory network B can output the target feature vector h of the first feature map using Equation 4. t :

[0131] Formula 4: h t =lstm(g t ,h t-1 )

[0132] Among them, h tLet g be the target feature vector, which is a one-dimensional vector. Let lstm() be a function of the recurrent memory network B. t The target enhancement vector for the first feature map.

[0133] It should be noted that the method of obtaining the first position data through attention network model A and recurrent memory network model B is similar to the method of obtaining the second position data. The following only describes in detail how to obtain the first position data through attention network model A and recurrent memory network model B:

[0134] The step of obtaining the position data (first position data) of the distribution of the multiple text regions in the first direction includes: performing pixel transformation and pixel adjustment processing on the target feature vector of the current cycle to obtain the transformation feature map of the current cycle; performing weighting processing on the first feature map according to the multiple similarity weights to obtain the weighted feature map; and performing fusion processing on the transformation feature map and the weighted feature map to obtain the second feature map of the current cycle, wherein the second feature map includes the position data of the distribution of the multiple text regions in the first direction.

[0135] In some embodiments, the second feature map of the current cycle can be obtained using Formula 5:

[0136] Formula 5: y t =α*F+nn.linear(h t ).reshape(H,W)

[0137] Among them, y t Let α*F be the second feature map, and nn.linear(h) be the weighted feature map. t ) represents the target feature vector h in the current iteration. t Perform pixel transformation processing, reshape(H,W) is an expression of nn.linear(h t Perform pixel adjustment processing on nn.linear(h) t The pixel features in the image are reordered to form a transformed feature map with a width of H and a height of W.

[0138] It should be noted that when the first direction is a column direction, the second feature map of the current loop is the image of a column of text regions highlighted. Assuming the target table image contains N columns of text regions, the attention-memory joint network model will loop N+1 times. In the first N loops, it outputs the highlighted text regions column by column in order from left to right, until the N+1th loop outputs a blank image, at which point the loop stops. For example, refer to... Figure 13The target table image contains 4 columns of text regions. The attention-memory joint network model loops 5 times. In the first loop (t=1), the first column of text regions is highlighted. In the second loop (t=2), the second column of text regions is highlighted. In the third loop (t=3), the third column of text regions is highlighted. In the fourth loop (t=4), the fourth column of text regions is highlighted. The loop continues until the fifth loop outputs a blank image.

[0139] Similarly, when the first direction is a row direction, the second feature map of the current loop is an image of a certain row of text regions highlighted. Assuming the target table image contains M rows of text regions, the attention-memory joint network model will loop M+1 times. In the first M loops, it outputs the highlighted images of the M rows of text regions line by line in descending order, until the (M+1)th loop outputs a blank image, at which point the loop stops. For example, if the target table image contains 3 rows of text regions, the attention-memory joint network model will loop 4 times. In the first loop, it outputs the highlighted image of the first row of text regions; in the second loop, it outputs the highlighted image of the second row of text regions; in the third loop, it outputs the highlighted image of the third row of text regions, until the fourth loop outputs a blank image.

[0140] It should also be noted that the weighted feature map α*F (see...) Figure 13 The bold black line in the image highlights all text regions in the first feature map. Using the weighted feature map as a reference for obtaining the second feature map significantly improves the recognition ability and accuracy of table recognition. In other words, by adding a weighted feature map α*F during the table structure extraction process, the table recognition effect is enhanced.

[0141] In summary, the table recognition method P100 and system 001 provided in this specification acquire a target table image containing a target table, wherein the target table includes multiple text regions. The table structure is extracted from the target table image based on the positions of the multiple text regions. The table structure includes position data of the multiple text regions distributed in a first direction and position data of their distribution in a second direction. Based on the table structure, the content data of the target table is extracted from the target table image, and the content data of the target table is output according to the table structure. The table recognition method of this application depends on the position data of the text regions and does not rely on the existence of dividing lines between the multiple text regions. Therefore, it can accurately recognize the target table image regardless of whether it is horizontally or vertically straight or distorted, and is not limited to wired or wireless table images. Moreover, the extraction of the table structure of the target table can be performed without recognizing the text in the target table, i.e., without the assistance of a text recognizer, directly restoring the table structure from the target table image, making it more end-to-end and thus simpler.

[0142] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing data processing. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the data processing method P100 described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 300, the program code causes the computing device 300 to perform the steps of the data processing method P100 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing device 300. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 300, partially on computing device 300, as a standalone software package, partially on computing device 300 and partially on a remote computing device, or entirely on a remote computing device.

[0143] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0144] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0145] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0146] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0147] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. The entire contents used for all purposes, except for any history of prosecution documents associated with it, that may be inconsistent with or conflict with this document, or that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0148] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A table recognition method, comprising: Obtain an image of a target table containing a target table, the target table including multiple text regions arranged along the row and / or column directions of the target table, each text region including at least one character; Obtain a two-dimensional first feature map of the target table image, wherein the first feature map includes multiple pixels; A first projection map of the first feature map is obtained in a first direction, and the first projection map is used as a reference for judging the text region along the first direction. The first projection map and the first feature map are input into the attention-memory joint network model, and the attention-memory joint network model performs a two-dimensional scan of the first feature map in pixels to obtain the position data of the distribution of the multiple text regions in the first direction. A second projection map of the first feature map is obtained in the second direction, and the second projection map is used as a reference for judging the text region along the second direction. The second projection map and the first feature map are input into the attention-memory joint network model, and the attention-memory joint network model performs a two-dimensional scan of the first feature map in pixels to obtain the position data of the distribution of the multiple text regions in the second direction. By combining the positional data of the multiple text regions distributed in the first direction and the positional data distributed in the second direction, the table structure of the target table is extracted; as well as Based on the table structure, the content data of the target table is extracted from the target table image, and the content data of the target table is output according to the table structure.

2. The method of claim 1, wherein the first direction is one of the row direction and the column direction, and the second direction is the other of the row direction and the column direction.

3. The method of claim 1, wherein obtaining the target table image containing the target table includes: Obtain an image to be processed, wherein the image to be processed is a scanned image of a target table printed on a medium, and the target table image in the image to be processed includes a distorted and tilted target table; The image to be processed is tilt-corrected to correct the tilt of the target table, resulting in a corrected image; and The corrected image is subjected to table detection, and the table region in the corrected image is extracted to obtain the target table image containing the target table.

4. The method of claim 1, wherein obtaining the first projection map of the first feature map in the first direction comprises: The first feature map is pooled in the first direction to obtain the first projection map; The step of obtaining the second projection map of the first feature map in the second direction includes: The first feature map is pooled in the second direction to obtain the second projection map.

5. The method of claim 1, wherein The step of obtaining the positional data of the distribution of the plurality of text regions in the first direction includes: The first feature map is input into a first attention-memory joint network model trained for the distribution in the first direction to calculate a first target feature vector pixel by pixel, and the position data of the distribution of the multiple text regions in the first direction is obtained based on the first target feature vector. The step of obtaining the positional data of the distribution of the multiple text regions in the second direction includes: inputting the first feature map into a second attention-memory joint network model trained for the distribution in the second direction to calculate a second target feature vector pixel by pixel on the first feature map, and obtaining the positional data of the distribution of the multiple text regions in the second direction based on the second target feature vector.

6. The method of claim 1, wherein the attention-memory joint network model includes an attention network model and a recurrent memory network model, wherein in the current iteration: The recurrent memory network model is configured to output the target feature vector of the first feature map and use the target feature vector as the input of its next iteration; The attention network model is configured to receive and, based on the first feature map and the target feature vector output by the recurrent memory network model in the previous iteration,... Multiple similarity weights, each corresponding one-to-one with the plurality of pixels, are determined and output. Each similarity weight describes the degree of similarity between each pixel in the first feature map and the target feature vector output in the previous iteration. The target enhancement vector of the first feature map is determined and input into the recurrent memory network model.

7. The method of claim 5, wherein the initial input value of the first attention-memory joint network model is the first projection map, the first projection map being a projection of the first feature map onto the first direction; and The initial input value of the second attention-memory joint network model is the second projection map, which is the projection of the first feature map onto the second direction.

8. The method of claim 6, wherein obtaining the positional data of the distribution of the plurality of text regions in the first direction includes: Pixel transformation and pixel adjustment are performed on the target feature vector of the current cycle to obtain the transformed feature map of the current cycle; The first feature map is weighted according to the multiple similarity weights to obtain a weighted feature map; as well as The transformed feature map and the weighted feature map are fused to obtain the second feature map of the current cycle. The second feature map includes the position data of the multiple text regions distributed in the first direction.

9. The method of claim 1, wherein extracting the content data of the target table from the target table image comprises: Perform character recognition on each of the multiple text regions included in the target table to obtain the characters within each text region; Obtain the row header and list header corresponding to each text region; as well as The text within each text area, the row header and the list header corresponding to each text area are determined as the content data in the target table.

10. A table recognition system, comprising: At least one storage medium storing at least one instruction set for table recognition; as well as At least one processor is communicatively connected to the at least one storage medium. When the table recognition system is running, the at least one processor reads the at least one instruction set and implements the table recognition method according to any one of claims 1-9.

Images