Information processing systems, information processing methods, and programs

Abstract

This invention provides an information processing system, information processing method, and program that extract item values ​​with high accuracy. [Solution] In an information processing system in which a management server is connected to a user terminal or user server via a communication network, the management server analyzes text from document image data of a document acquired from a user device, analyzes the layout of the document from the document image data, analyzes the reading order of the document from the document image data, and analyzes the table structure contained in the document from the document image data, and extracts the values ​​of items contained in the document image data based on one or more analyses of the text, the document layout, the reading order, and the table structure.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background Art]

[0002] Conventionally, so-called OCR technology, which performs character recognition from document image data to extract information from document data and the like, has been widely used. For example, Patent Document 1 discloses reading characters from a form. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2002-109468 [Summary of the Invention] [Problems to be Solved by the Invention]

[0004] However, although the system described in Patent Document 1 can accurately identify existing forms and perform character recognition, in recent years, there has been an increasing demand for recognition of items composed of characters, numbers, and the like for document data having various formats, and the demand for recognition accuracy is also increasing.

[0005] The present invention has been made in view of such a background, and an object of the present invention is to provide a highly accurate item value recognition technology that can support various document formats. [Means for Solving the Problems]

[0006] One embodiment of the main invention for solving the above problems is a user device and a management server connected to the user device via a network, wherein the management server analyzes text from document image data of a document obtained from the user device, analyzes the layout of the document from the document image data, analyzes the reading order of the document from the document image data, and / or analyzes the table structure contained in the document from the document image data, and extracts the values ​​of items contained in the document image data based on one or more analyses of the text, the layout of the document, the reading order, and / or the table structure.

[0007] Further issues and solutions disclosed in this application will be made clear in the section on embodiments of the invention and in the drawings. [Effects of the Invention]

[0008] According to the present invention, the values ​​of items can be extracted with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of an information processing system. [Figure 2] This figure shows an example of the hardware configuration of management server 2. [Figure 3] This figure shows an example of the software configuration for management server 2. [Figure 4] This diagram illustrates the operation of management server 2. [Figure 5] This is a conceptual diagram illustrating the recognition of layout and reading order. [Figure 6] This is a conceptual diagram illustrating the recognition and output of table structures. [Figure 7] This diagram illustrates other operations of the management server 2. [Figure 8] This diagram illustrates further operations of management server 2. [Figure 9] This diagram illustrates further operations of management server 2. [Figure 10]It is a conceptual diagram illustrating character string recognition. [Figure 11] It is a diagram illustrating still another operation of the management server 2. [Figure 12] It is a diagram illustrating still another operation of the management server 2. [Figure 13] It is a conceptual diagram illustrating table structure analysis. [Figure 14] It is a conceptual diagram illustrating table structure analysis. [Figure 15] It is a conceptual diagram illustrating table structure analysis.

Mode for Carrying Out the Invention

[0010] <Outline of System> Hereinafter, an information processing system according to an embodiment of the present invention will be described. The information processing system of the present embodiment is intended to perform OCR processing for reading values of one or more items composed of character strings and / or numerical values from document data. In the information processing system of the present embodiment, one or a combination of character recognition, layout recognition, and reading order recognition technologies is applied to items from document data such as PDF data and image data according to the document data, thereby achieving highly accurate reading of item values.

[0011] FIG. 1 is a diagram showing an example of the overall configuration of the information processing system. The information processing system of the present embodiment is configured including a management server 2. The management server 2 is communicably connected to a user terminal 1 or a user server 3 via a communication network. The communication network is, for example, the Internet, and is constructed by a public telephone line network, a mobile telephone line network, a wireless communication path, Ethernet (registered trademark), or the like.

[0012] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, a personal computer, or the like.

[0013] The management server 2 and user server 3 may be general-purpose computers such as workstations or personal computers, or may be logically realized by cloud computing.

[0014] <Management Server> Figure 2 is a diagram showing an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and the management server 2 may have other configurations. The management server 2 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid-state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for performing wireless communication, or a USB (Universal Serial Bus) connector or RS232C connector for serial communication. The input device 205 is for inputting data, and is, for example, a keyboard, a mouse, a touch panel, buttons, or a microphone. The output device 206 is for outputting data, and is, for example, a display, a printer, or a speaker. It should be noted that each functional unit of the management server 2 described later is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing the program, and each storage unit of the management server 2 is realized as a part of the storage area provided by the memory 202 and the storage device 203.

[0015] Figure 3 is a diagram showing an example of the software configuration of the management server 2. The management server 2 includes an AI model storage unit 231, an analysis data storage unit 232, an image data acquisition unit 211, a text analysis unit 212, a layout analysis unit 213, a reading order analysis unit 214, a table structure analysis unit 215, and a data output unit 216.

[0016] <Storage Unit> The AI ​​model memory unit 231 stores the weight parameters of four types of learning models, each trained using a predetermined training dataset on a server outside the management server 2, for example. In this embodiment, using a deep learning-based method, the weight parameters optimized for each of the following models can be stored: a learning model enhanced to extract automatically learned vector data as features and acquire text data from document data (text recognition learning model); a learning model enhanced to extract item values ​​consisting of characters, numbers, and symbols from acquired text data (text detection learning model); a learning model enhanced to understand document layouts such as vertical writing and column layouts from document data, acquire layout data, predict reading order, and extract item values ​​without distorting the semantic structure due to the layout (layout analysis learning model); and a learning model enhanced to understand table structures from document data and recognize table structures including complex tables without cell merges or separator lines (table structure analysis learning model). In this embodiment, the learning models are particularly characterized by providing learning models that are specialized for Japanese documents and have enhanced capabilities for acquiring text data, extracting item values, acquiring layout data, and acquiring reading order data.

[0017] Furthermore, the analysis data storage unit 232 stores the values ​​of the items extracted based on the learning model as analysis data. The analysis data is initially stored in a predetermined format such as CSV or Excel, but it can be converted to a format specified by the user (e.g., HTML, Markdown, JSON, etc.), stored as analysis data, and output according to the user's request.

[0018] <Functional Section> The image data acquisition unit 211 acquires image data (hereinafter referred to as document image data) of the document to be analyzed (the document from which items should be extracted). The image data acquisition unit 211 can receive document image data from the user terminal 1. The image data acquisition unit 211 may also acquire document image data by converting any format, such as a PDF file or a presentation file, into an image. In addition, when the management server 2 is operating independently, the image data acquisition unit 211 may acquire document image data captured by a camera or document image data scanned by a scanner.

[0019] The text analysis unit 212 obtains text data obtained by analyzing document image data. Here, the text data analysis unit 212 recognizes the positional region of text and the strings present within that region as text data from the image based on the learning model, and can detect the values ​​of items consisting of characters, numbers, and symbols from the recognized text data. In text detection, the image is used as input data, the learning model extracts feature vectors from the image, and predicts a numerical value representing the probability of a character existing for each pixel in the entire image from these feature vectors. Thresholding is applied to these numerical values ​​to separate them into character regions and background regions. Next, the contour is obtained for the concatenated character regions, and the smallest rectangular region encompassing the contour is calculated. The region with a margin added to this smallest rectangular region is estimated as the region where characters exist. Here, the learning model for text detection is implemented by supervised learning, using a dataset in which string regions of document images have been labeled in advance to generate pixel-level training labels for the image. Here, for vertical text layout recognition, a projection transformation is performed on the rectangular region detected by the text detection learning model described above to convert it into a rectangular image. Then, the aspect ratio of the rectangular image is referenced, and the vertically elongated image is rotated 90 degrees counterclockwise. The above image is input into a text recognition learning model. While considering the correlation between image feature vectors, the model predicts string tokens by taking into account not only the image information but also the surrounding and overall linguistic information, and then decodes them into characters corresponding to the tokens.

[0020] Here, the text recognition learning model collects Japanese and English word and sentence corpora in advance and uses them as training data. Therefore, it learns and considers not only the visual information of the image but also the linguistic meaning before predicting the string. The text recognition model prepares images in which the string region has been extracted from the document image and training labels in which the strings present in the string region have been transcribed, and uses these for training.

[0021] The layout analysis unit 213 analyzes the layout of the document image data based on the learning model described above. The layout analysis unit 213 recognizes the layout of the document from the document image data, based on the learning model described above, including the complex document structure, such as vertical writing and column layouts, especially for Japanese texts, as shown in Figure 5(a). Specifically, it estimates the position coordinates and areas of elements that make up the document, such as headings, paragraphs, tables, images, bluffs, and headers, from within the document image. The learning model for layout analysis constructs training data in which the position of each document component associated with an image is labeled in advance. Based on this information, it learns the ability to predict the coordinates of document components from a document using supervised learning.

[0022] The reading order analysis unit 214, based on the layout analysis results obtained from the above analysis of document image data and the document layout recognized based on the learning model for the above layout analysis, infers the order in which a human would read the document, as shown in Figure 5(b), and recognizes the reading order of the document without distorting the semantic structure of the layout. Specifically, it calculates the optimal path that traverses all document components for the document image components obtained by the layout analysis. An adjacency graph is constructed in advance for adjacent document component regions, and a rule-based prioritized depth-first search is performed on the adjacency graph to calculate a path that traverses all document components.

[0023] The table structure analysis unit 215 analyzes the table image contained in the document image data based on the learning model described above, and recognizes the table structure contained in the document, including complex tables without cell merges or separator lines, as shown in Figure 6(a). For example, as shown in Figure 6(b), it can recognize the values ​​of items in any format. Specifically, table structure analysis is performed by obtaining partial images of the detected table image regions through layout analysis and applying it to those images. The table image is input, and the table structure analysis learning model estimates the rows, columns, cell merges, column header regions, and row header regions of the document. The rows and columns are sorted by coordinates, and the numbers of each row and column are estimated. Next, the overlapping regions where each row and column region intersect are calculated, and the target row and column numbers are assigned to these overlapping regions = cell regions. Then, cell regions enclosed by cell merge regions are inspected, and if multiple cell regions exist within a cell merge region, the cell regions are merged. The merged cell region is assigned the smallest column and row numbers, and the information on the largest column and row numbers of the merged cell region is taken into consideration to predict how many rows and columns the merged region spans. The learning model for table structure analysis assigns information such as columns, rows, cell merges, and header areas to table images as training labels, and then obtains an optimized model through supervised learning based on that information.

[0024] The data output unit 216 extracts the values ​​of items (e.g., strings, numbers, etc.) composed of characters, numbers, and / or symbols from the document image data based on the analysis results of the text, layout, reading order, and / or table structure, and outputs them to the user device 1 in a predetermined format. At this time, the data output unit 216 can also store the values ​​of the items as structured data in a predetermined format (e.g., CSV, Excel, etc.) in the analysis data storage unit 232, and / or convert the data to a format specified by the user (HTML, Markdown, JSON, etc.), store it in the converted format, and output it to the user device 1. Here, the data output unit 216 can extract items as item values ​​that are included in the document data and recognized based on the above analysis, items that are pre-set in the management server 2, or items that are specified by the user and received from the user device 1.

[0025] <Operation> Figure 4 is a diagram illustrating the operation of the management server 2.

[0026] The management server 2 acquires document image data (S101), recognizes text from the document image data (S102), recognizes the layout from the document image data (S103), recognizes the reading order based on the layout from the document image data (S104), recognizes the table structure from the document image data (S105), and extracts the values ​​of items contained in the document image data based on the analysis results of the text, layout, reading order and / or table structure (S106). The management server 2 outputs the extracted item values ​​in a predetermined format and stores them as analysis data in the analysis data storage unit 232.

[0027] Here, the management server 2 can determine which learning model is suitable for which type of analysis—text analysis, layout analysis, reading order analysis, or tabular structure analysis—based on the acquired document image data, and then analyze the document image data based on the learning model suitable for one of the above analyses to extract the values ​​of the items.

[0028] Figure 7 illustrates other operations of the management server 2.

[0029] The management server 2 refers to the above analysis data (S201), converts it into data in a format specified by the user device 1 (S202), and outputs the converted analysis data to the user device 1 by sending it to the user device 1 (S203). In addition, the management server 2 can also output the analysis data to the user device 1 in a default format (e.g., CSV, Excel, etc.), with or without user instruction.

[0030] Figure 8 illustrates another operation of the management server 2 (string detection processing).

[0031] Management Server 2 acquires the document image (S301), normalizes the pixel values ​​of the document image (S302), predicts the probability that a string exists for each pixel using a learning model (S303), binarizes the predicted probabilities using a threshold, and separates them into a string region and a background region (S304). Management Server 2 acquires the contour line encompassing the string region (S305) and calculates the smallest rectangular region encompassing the contour line of the string region (S306). Then, it adds a margin region to the smallest rectangular region and expands the rectangular region (S307).

[0032] Figure 9 illustrates another operation of the management server 2 (string recognition processing).

[0033] The management server 2 acquires the string area (S401), performs a projection transformation on the string area as shown in Figure 10 to convert it into a rectangular image (S402), and rotates it by 90 degrees relative to the vertical image by referring to the aspect ratio of the rectangular image (S403). Then, the management server 2 normalizes the pixel values ​​of the string image (S404), predicts the character tokens contained in the string image using a learning model (S405), and decodes the character tokens into the assigned character codes (S406).

[0034] Figure 11 illustrates another operation (layout analysis process) of the management server 2.

[0035] Management Server 2 acquires character images (S501), normalizes the pixel values ​​of the character images (S502), and estimates paragraphs, headings, figures, and table areas using a learning model (S503). Then, Management Server 2 extracts and acquires partial images of the predicted table and figure areas (S504).

[0036] Figure 12 illustrates another operation of the management server 2 (table structure analysis processing).

[0037] The management server 2, in the layout analysis unit 213, acquires the table area (S601), normalizes the pixel values ​​of the character image (S602), and acquires the table image data (S603). Then, as shown in Figures 13 and 14, the management server 2 estimates the columns, rows, cell merges, column headers, and row header areas using a learning model (S604), sorts the row and column areas by coordinate position, and assigns row and column numbers (S605). Then, as shown in Figure 15, it calculates the overlapping areas where the row and column areas intersect and acquires individual cell areas (S606). Subsequently, as shown in Figure 15, it inspects the cells included in the cell merge area and merges the cells included in the cell merge area (S607). Furthermore, it inspects the cells included in the row header and column header areas and determines that the included cells are headers (S608).

[0038] As described above, the information processing system of this embodiment can extract item values ​​with high accuracy from document image data of various formats through multifaceted analysis including text, layout, reading order, and table structure.

[0039] Although these embodiments have been described above, they are intended to facilitate understanding of the present invention and are not intended to limit its interpretation. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are also included.

[0040] Furthermore, the user device 1 or the user server 3 managed by the user may store the information stored in each storage unit of the management server 2 and the programs that execute the processing performed by each functional unit, as described above, so that the entire process can be executed within the user device 1 or user server 3. This allows for faster processing on the user device 1 or user server 3, and since data such as document data and templates only needs to be uploaded to the user device 1 or user server 3, it can be executed securely.

[0041] Furthermore, for example, the processing performed by each functional unit of the management server 2 described above may be executed by any of the functional units. Additionally, different functional units may be added to perform some of the processing performed by each of the functional units described above. Furthermore, the functional units of the management server 2 may be distributed across multiple computers.

[0042] Furthermore, the information stored in each memory unit of the management server 2 may be stored in any of the memory units. That is, the information stored in the multiple memory units mentioned above may be stored in a single memory unit, or a portion of the information stored in one memory unit may be stored in another memory unit. [Explanation of Symbols]

[0043] 1 User terminal 2 Management Server 3. User Server

Claims

1. A user device and a management server connected to the user device via a network, The aforementioned management server The text is analyzed from the document image data of the document acquired from the user device. From the aforementioned document image data, the layout of the document is analyzed. From the document image data, the reading order of the document is analyzed, and / or, From the aforementioned document image data, the table structure contained in the document is analyzed, A management server that extracts values ​​of items included in the document image data based on one or more analyses of the text, the document layout, the reading order, and / or the table structure.

2. A management server according to claim 1, The management server performs the one or more analyses based on a learning model that enhances the analysis of the text, a learning model that enhances the analysis of the layout and reading order, and a learning model that enhances the analysis of the table structure.

3. A management server according to claim 1, The aforementioned learning model is a management server generated by learning template data acquired from user devices.

4. A management server according to claim 1, A management server that outputs the values ​​of the extracted items as analysis data in the format specified by the user device.

5. A user device and a management server connected to the user device via a network, On the aforementioned management server, The text is analyzed from the document image data of the document acquired from the user device. From the aforementioned document image data, the layout of the document is analyzed. From the document image data, the reading order of the document is analyzed, and / or, From the aforementioned document image data, the table structure contained in the document is analyzed, A program for causing a computer to extract values ​​of items contained in the document image data based on one or more analyses of the text, the document layout, the reading order, and / or the table structure.

Images