Image processing system, and image processing method, trained model and training method for the same, and program

Abstract

To provide an image processing system and an image processing method, and a trained model and a training method for the same and a program for generating training data with high accuracy and high productivity by using data indicating the boundary between cells as a correct answer label.SOLUTION: A method in an image processing system uses a trained model through machine learning using a document image including a table as input and the boundary between cells of the table included in the input document image as output, inputs the document image to the trained model to acquire the boundary between the cells of the table included in the document image, and specifies a cell regions of the table from the boundary between the cells of the table.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an image processing method, a trained model, a training method thereof, and a program. [Background technology]

[0002] Conventionally, documents such as invoices and purchase orders include schedules containing information such as the product names, quantities, unit prices, and amounts of goods and services related to billing and ordering. There are technologies for recognizing table structures to extract information contained in such schedules. For example, in Patent Document 1, machine learning is used on an image containing a schedule to recognize each cell area in the table by determining whether each pixel belongs to a table cell on a pixel-by-pixel basis. Furthermore, optical character recognition (OCR) processing is performed on the image, and the acquired character string is linked to each cell area. This allows, for example, the extraction of item values ​​such as the quantity, unit price, and amount for a particular product by recognizing table rows based on the position of each cell area and the character string acquired by OCR processing. Furthermore, by recognizing table columns based on the position of each cell area and the character string acquired by OCR processing, a list of product names can be extracted from the product name column. In this way, information contained in a schedule can be extracted. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 1,108,7123 Summary of the Invention [Problem to be solved by the invention]

[0004] As in Patent Document 1, learning data (also called teacher data) is required to create a model that recognizes each cell area in a table pixel-by-pixel using machine learning. The learning data consists of a document image containing a schedule and a correct label image indicating each cell area in the table. Of these, the correct label image is created manually. For example, as shown in FIG. 11(a), a document image 1101 is displayed on a screen 1100 for creating correct labels, and a rectangular area on the image, such as area 1102, is created for each cell area in the table. The area is created by dragging the mouse cursor 1103 using a mouse or the like. Once the creation is complete, a correct label image 1110, such as that shown in FIG. 11(b), is obtained by pressing a save button 1104. Thus, creating a correct label image requires the user to specify as many areas as there are cell areas in the table, which is cumbersome. Furthermore, when specifying an area, the left and right edges of the area are likely to be misaligned, as in area 1111 in FIG. 11(b), or the top and bottom edges are likely to be misaligned, as in area 1112. Therefore, careful operation was required to avoid such misalignment when creating the correct labels, which was inefficient. Misalignment of the left and right or top and bottom edges of each cell area resulted in inconsistent training data, and it was difficult to obtain good results with models obtained using such training data.

[0005] The present invention has been made in consideration of the above-mentioned conventional examples, and aims to create training data with high accuracy and high productivity by using data indicating boundaries between cells as correct labels. [Means for solving the problem]

[0006] In order to achieve the above object, according to one aspect of the present invention, there is provided an image processing system using a trained model based on machine learning, which receives an input of a document image including a table and outputs the boundaries of cells of the table included in the input document image, an acquisition means for inputting a document image into the trained model and acquiring boundaries of table cells included in the document image; a first specifying means for specifying a table cell area from the boundaries of the table cells; An image processing system is provided.

[0007] According to another aspect of the present invention, there is provided a trained model by machine learning that receives an input of a document image including a table and outputs table cell boundaries included in the input document image, The computer acquiring training data in which a training document image and a training image in which boundaries of table cells included in the training document image are identified are associated with each other; a step of inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; Obtaining a loss function value using the cell boundaries output by the learning model; optimizing the learning model so that the value of the loss function becomes small; and The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. A trained model is provided, characterized by: [Effects of the Invention]

[0008] According to the present invention, by using data indicating boundaries between cells as correct labels, training data can be created with high accuracy and high productivity. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating the configuration of an information processing device, a learning device, and an information processing server. [Figure 3] FIG. 10 is a diagram showing a usage sequence of the information processing system. [Figure 4] 10 is a flowchart showing a flow of generating a cell boundary recognition model. [Figure 5]10 is a flowchart showing a flow of extracting item values ​​from a document image. [Figure 6A] 10A and 10B are diagrams showing specific examples of learning document images and correct label images; [Figure 6B] 10A and 10B are diagrams showing specific examples of learning document images and correct label images; [Figure 7] FIG. 10 is a diagram showing a screen for creating a correct label image of a cell boundary in this embodiment. [Figure 8] 10A and 10B are diagrams illustrating specific examples of recognition results of a cell boundary recognition model. [Figure 9] 10A and 10B are diagrams illustrating a specific example of extracting item values ​​based on the cell boundary recognition results. [Figure 10] FIG. 10 is a diagram showing a specific example of a confirmation screen for extracted item values. [Figure 11] FIG. 10 is a diagram showing a screen for creating a correct label image of a cell region. [Figure 12] 10 is a flowchart showing the flow of generating a cell boundary recognition model in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] <Information Processing System> FIG. 1 is a diagram showing an example configuration of an information processing system or an image processing system. As shown in FIG. 1, the information processing system 100 is composed of, for example, an information processing device 101, a learning device 102, and an information processing server 103, which are connected to each other via a network 104. In the information processing system 100, each of the information processing device 101, the learning device 102, and the information processing server 103 may not be a single device, but may be a configuration in which multiple devices are connected. For example, the information processing server 103 may be composed of a first server device having high-speed computing resources and a second server device having large-capacity storage, which are connected to each other via the network 104.

[0012] The information processing device 101 is realized by a multi-function peripheral (MFP) having multiple functions such as printing, scanning, and faxing, and functions as an image acquisition unit 1011. The information processing device 101, for example, optically reads an original 111 printed on a storage medium such as paper using the image acquisition unit 1011, performs predetermined scan image processing to generate a document image 113, and transmits the document image 113 to the information processing server 103. The information processing device 101 also receives, for example, FAX data 112 transmitted from a facsimile (FAX) transmitter (not shown), performs predetermined FAX image processing to generate a document image 113, and transmits the document image 113 to the information processing server 103. Note that the information processing device 101 may be realized by a personal computer (PC) or the like, in addition to the MFP having the above-mentioned scanning and faxing functions. Specifically, for example, a document image 113 such as PDF or JPEG generated using a document creation application running on a PC serving as the information processing device 101 may be transmitted to the information processing server 103 .

[0013] The learning device 102 functions as a generation unit 1021 that acquires multiple training document images 114 and generates corresponding correct label images 115. The learning device 102 also functions as a learning unit 1022 that uses the training document images 114 and the correct label images 115 generated by the generation unit as teacher data or training data to learn and generate a cell boundary recognition model 116 as a trained model. Here, the "cell boundary recognition model" is a recognizer that recognizes table boundaries in document images. The cell boundary recognition model is configured as a neural network. "Learning" in the above context refers to adjusting the parameter values ​​of the cell boundary recognition model, which is a neural network, using a deep learning method based on the training document images 114 and the correct labels 115. The cell boundary recognition model can be trained using any known method. For example, a fully convolutional network (FCN) that detects objects in an image at the pixel level can be used. A model that has completed learning is also called a trained model, and a model currently being trained is also called a learning model.

[0014] A neural network is a machine learning technique that consists of, for example, an input layer, one or more intermediate layers, and an output layer. In a typical example of such a configuration, each layer has multiple nodes, and the nodes are connected to one or more other nodes. Each node has parameters such as weights, and after weighting the input data, it transmits it to the node in the next layer. This data propagation occurs from the input layer where the data is input to the intermediate layer and then to the output layer, resulting in an output. In a neural network, parameters such as weights at each node are adjusted through learning to obtain a trained model. During learning, the error between the output and the correct answer is evaluated using a loss function, and the error is reflected in the parameters using, for example, the backpropagation method, updating the parameters and progressing the learning. Once a trained model that reduces the error below a predetermined value has been created, data can be input to the input layer as described above to obtain the processing results from the output layer.

[0015] The information processing server 103 functions as an information processing unit 1031 that acquires multiple character strings contained in a document image 113 received from the information processing device 101 and extracts character strings 117 from the statement, and as a storage unit 1032 that stores the extracted results. First, the information processing unit 1031 of the information processing server 103 performs OCR processing on the document image 113 and acquires multiple character strings as character recognition results obtained by the OCR processing. Furthermore, the information processing unit 1031 of the information processing server 103 recognizes boundaries between cells in the statement from the document image 113 using the cell boundary recognition model 116, which is a trained model received from the learning unit 1022 of the learning device 102. The information processing unit 1031 further detects closed areas surrounded by boundaries as cells and extracts character strings from each cell by OCR processing or the like. Here, the extracted character strings 117 from the statement are those written in the statement, such as the product name, quantity, unit price, and amount.

[0016] The network 104 is realized by a local area network (LAN), a wide area network (WAN), etc., and is a communication medium that connects the information processing device 101, the learning device 102, and the information processing server 103 to each other and transmits and receives data between the devices.

[0017] <Device configuration> FIG. 2 is a diagram showing an example of the configuration of an information processing device 101, a learning device 102, and an image processing server 103 for realizing the information processing system 100 of FIG.

[0018] 2(a) is a diagram showing the configuration of an information processing device 101. As shown in FIG. 2(a), the information processing device 101 may be an image forming device called an MFP or the like, and has the following configuration: It has a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, and a document transport device 207. It also has a storage 208, an input device 209, a display device 210, and an external interface 211, which are connected to each other via a data bus 203.

[0019] The CPU 201 is a control unit for controlling the overall operation of the information processing device 101. The CPU 201 starts up the system of the information processing device 101 by executing a startup program stored in the ROM 202. The CPU 201 also executes a control program stored in a storage unit such as the storage 208 to realize functions of the information processing device 101 such as printing, scanning, and faxing.

[0020] The ROM 202 is realized by a non-volatile memory, and is a storage unit for storing a boot program for booting the information processing device 101.

[0021] The data bus 203 is a communication unit for transmitting and receiving data between devices that make up the information processing device 101 .

[0022] The RAM 204 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 201 executes a control program.

[0023] The printer device 205 is an image output device, and is a processing unit for printing and outputting a document image inside the information processing device 101 onto a storage medium such as paper.

[0024] The scanner device 206 is an image input device and a processing unit for optically reading a storage medium such as paper on which characters, diagrams, and the like are printed, and acquiring the result as a document image.

[0025] The document transport device 207 is realized by an ADF (Auto Document Feeder) or the like, and is a processing unit for detecting documents placed on a document table and transporting the detected documents one by one to the scanner device 206.

[0026] The storage 208 is realized by a hard disk drive (HDD) or the like, and is a storage unit for storing the control programs and document images described above.

[0027] The input device 209 is realized by a touch panel, hard keys, or the like, and is a processing unit for accepting operation inputs to the information processing device 101 from the user.

[0028] The display device 210 is realized by a liquid crystal display or the like, and is a display unit for displaying and outputting various display screens of the information processing device 101 to the user.

[0029] The external interface 211 connects the information processing device 101 to the network 104, and is an interface unit for receiving fax data from a fax transmitter (not shown) and transmitting document images to the information processing server 103.

[0030] 2(b) is a diagram showing the configuration of the learning device 102. The hardware of the learning device 102 may be a general-purpose computer. As shown in FIG. 2(b), the learning device 102 is composed of a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239, which are connected to each other via a data bus 233.

[0031] The CPU 231 is a control unit for controlling the overall operation of the learning device 102. The CPU 231 starts up the system of the learning device 102 by executing a boot program stored in the ROM 232. Then, the CPU 231 executes a learning program stored in the storage 235 to learn using learning data (teacher data) as input, thereby generating a cell boundary recognition model.

[0032] The ROM 232 is realized by a non-volatile memory, and is a storage unit for storing a boot program that starts up the learning device 102.

[0033] The data bus 233 is a communication unit for transmitting and receiving data between the devices that make up the learning device 102 .

[0034] The RAM 234 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 231 executes the learning program.

[0035] The storage 235 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the control program and cell boundary recognition model described above. The input device 236 is realized by a mouse, keyboard, or the like, and is a processing unit for accepting operation inputs to the learning device 102 from an engineer (an operator of the learning device 102).

[0036] The display device 237 is realized by a liquid crystal display or the like, and is a display unit for displaying and outputting various display screens of the learning device 102 to the engineer (user).

[0037] The external interface 238 connects the learning device 102 to the network 104, and is an interface unit for receiving images from a PC or the like (not shown) and transmitting the cell boundary recognition model 116 to the information processing server 103.

[0038] The GPU 239 is a calculation unit configured with an image processing processor. The GPU 239 executes calculations for training the cell boundary recognition model 116 based on training data, for example, in accordance with control commands given from the CPU 231.

[0039] 2(c) is a diagram showing the configuration of the information processing server 103. As shown in FIG. 2(c), the information processing server 103 is composed of a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268, which are connected to each other via a data bus 263.

[0040] The CPU 261 is a control unit for controlling the overall operation of the information processing server 103. The CPU 261 starts up the system of the information processing server 103 by executing a boot program stored in the ROM 262, and executes information processing such as character recognition and information extraction by executing an information processing program stored in a storage unit such as the storage 265. The CPU 261 also executes the information processing program and extracts cell boundaries included in a document image using a cell boundary recognition model generated by the learning device 102. Furthermore, for example, the CPU 261 may recognize characters within cells, save the recognition results, and output them to the user. Alternatively, the meaning of the recognized characters may be further identified, and the recognition results may be linked to the meaning and saved or output to the user. Furthermore, a title for the document image including the date, amount, client, etc. may be created based on the recognized character string and its meaning, and saved together with the document image.

[0041] The ROM 262 is realized by a nonvolatile memory, and is a storage unit for storing a boot program that starts up the information processing server 103.

[0042] The data bus 263 is a communication unit for transmitting and receiving data between devices that make up the information processing server 103 .

[0043] The RAM 264 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 261 executes an information processing program.

[0044] The storage 265 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the information processing program, the document image 113, the cell boundary recognition model 116, the character string data 117, and the like.

[0045] The input device 266 is realized by a mouse, keyboard, etc., and is a processing unit for accepting operational input to the information processing server 103 from a user who uses the information processing server 103 or an engineer who controls the information processing server 103.

[0046] The display device 267 is realized by an LCD display or the like, and is a display unit for displaying and outputting various display screens of the information processing server 103 to users who use the information processing server 103 or to engineers who control the information processing server 103.

[0047] The external interface 268 is an interface unit that connects the information processing server 103 and the network 104 and receives the cell boundary recognition model 116 from the learning device 102 and the document image 113 from the information processing device 101.

[0048] <Usage sequence> Fig. 3 is a diagram showing a usage sequence of the information processing system 100 in Fig. 1. Fig. 3(a) is a diagram explaining the flow of generating a cell boundary recognition model by the information processing system 100. In the following explanation, the processing by each device is realized by the CPU of each device executing a program stored in memory.

[0049] In step S301, an engineer developing the information processing system 100 inputs the learning document image 114 to the learning device 102.

[0050] In step S302, the engineer operates the learning device 102 to cause the generation unit 1021 to generate a correct answer label image 115 corresponding to the learning document image input in S301.

[0051] In step S303, the learning device 102 learns a model for recognizing table cell boundaries in document images based on the training data obtained in steps S301 and S302, which associates the training document images with correct labeled images (also called training images). Note that steps S301 to S303 may be repeated for a large number of training document images in order to improve the accuracy of the cell boundary recognition model.

[0052] In step S304, the learning device 102 transmits the cell boundary recognition model learned in S303 to the information processing server 103 as a learned model.

[0053] In step S305, the information processing server 103 stores the trained model received in S304 in the storage 265.

[0054] FIG. 3(b) is a diagram illustrating a flow in which the information processing system 100 outputs item values ​​included in the table of the document image 113 in response to a request from a user.

[0055] In step S311, a user of the information processing system 100 places an original 111 on the information processing device 101 and instructs the information processing device 101 to scan the original in order to acquire character strings within a document.

[0056] In step S312, the information processing apparatus 101 executes a scan process to generate a document image 113.

[0057] In step S313, the information processing apparatus 101 transmits the document image 113 generated in step S312 to the information processing server 103.

[0058] In step S314, the information processing server 103 receives the document image 113 transmitted in S313. Then, the information processing server 103 uses the cell boundary recognition model to recognize the boundaries between cells on the document image 113. Then, the information processing server 103 identifies the cell area surrounded by the recognized boundaries and extracts the character strings in the cells.

[0059] In step S315, the information processing server 103 outputs the character string extracted in S314 to the user. Note that the information processing server 103 may perform further processing using the character string extracted in S314 and the cell boundary recognized in S314.

[0060] <Learning methods> Fig. 4 is a flowchart illustrating the process of the learning device 102 learning a cell boundary recognition model in the sequence of Fig. 3(a). Note that the execution program for each step shown in Fig. 4 is stored in the ROM 232, RAM 234, or storage 235 of the learning device 102, and is executed by the CPU 231 of the learning device 102. The CPU 231 can execute part of the process using the GPU 239.

[0061] First, in step S401, the CPU 231 acquires the learning document image 114 and the correct label image 115 input by the engineer as learning data. The learning document image 114 is, for example, a document image 600 as shown in FIG. 6A(a), and the area indicated by the dashed line in the image is a schedule 601. The document image is, for example, a full-color (RGB three-channel) image with a size equivalent to A4 300 dpi (width 2480 pixels, height 3508 pixels). The correct label image 115 is, for example, an image 610 as shown in FIG. 6A(b), in which labels are assigned to pixels corresponding to boundaries between cells in the schedule. The correct label image is, for example, a binary image with the same image size as the document image. For pixels belonging to boundaries between cells, a value indicating a boundary (e.g., 255, indicating a white portion in the image) is set; otherwise, a value indicating a non-boundary (e.g., 0, indicating a black portion in the image) is set. The correct labeled image 610 is created, for example, as follows.

[0062] As shown in FIG. 7(a), the learning device 102 displays a document image 600 on a screen 700 for creating a correct label. Then, the engineer uses the input device 236 to input a cell boundary 702. This operation involves dragging the mouse cursor 703 in a straight line, as shown in the figure. In this case, the input line may be a line connecting the start point and end point of the drag operation. Alternatively, to draw a line accurately, the line may be specified by clicking the start point and end point, respectively. In either case, a rectangular area of ​​a predetermined width, with the line connecting the specified start point and end point as its center in the width direction, may be specified as the cell boundary. Note that the line width may also be specified separately. The operator specifies the cell boundaries included in the document image 600 using this operation. Then, for example, the operator initializes all pixels of a binary image of the same size as the document image 600 to 0, and rewrites the values ​​of pixels belonging to the boundary specified in the above manner to 255. This is an example of a case where each pixel is expressed with 8 bits (256 colors), but any number of colors can be expressed in binary in the same way.

[0063] When the creation is complete, the user presses the save button 704, and the correct label image 610 is obtained and saved as shown in FIG. 7(b).

[0064] In this way, correct labels can be created by simply drawing vertical and horizontal lines in the table areas corresponding to the cell boundaries. This reduces the number of operations and allows for more efficient creation of correct labels compared to the conventional method of setting rectangular cell regions. A detailed statement, such as an invoice, may contain information about one or more transaction objects. Information about each transaction object may include product names, quantities, unit prices, and amounts. For example, if three transaction objects are listed, the statement will have four rows and four columns, containing a total of 16 cells. In this case, while the conventional technology required the setting of 16 rectangular cell regions, the present invention only requires the setting of 10 linear boundaries, reducing the number of operations from 16 to 10. Furthermore, while the conventional technology required careful manipulation to align the edges of the rectangles, the present invention eliminates this by setting linear boundaries. These techniques are effective in creating the large amounts of training data (correct labels) required for neural network training.

[0065] While the detailed table 601 in the training document image 600 is a table with explicit ruled lines drawn at the boundaries between cells, images containing other tables may also be used as training document images. For example, there is a table, such as detailed table 620 shown in FIG. 6B(c), which has no ruled lines and alternates between rows with and without background color. The correct label image for this table is image 630 shown in FIG. 6B(d). That is, correct labels are created by considering the boundaries of background colors and spaces between character strings that engineers recognize as cell boundaries. There is also a table, such as detailed table 640 shown in FIG. 6B(e), which has no ruled lines and no background color. The correct label image for this table is image 650 shown in FIG. 6B(f), and correct labels are created at the locations that the engineer, the operator, recognizes as cell boundaries based on the arrangement of character strings, etc. Also, there are tables in which an entire row is combined into one cell, such as the detailed table 660 shown in Fig. 6B(g) (a row in the table that contains character strings indicating titles for listing by product category, such as "office supplies" or "peripheral devices"). The correct label image for this table is image 670 shown in Fig. 6B(h), and when drawing a line vertically, the correct label is created excluding cells that are one entire row.

[0066] 4, in step S402, the CPU 231 initializes the parameters of the cell boundary recognition model. The CPU 231 randomly determines and initializes the values ​​of the parameters included in the model. Note that other methods for determining the values ​​used for initialization may also be used. For example, parameter values ​​in a model previously trained using training document images and supervised label images different from the training document images 114 and supervised label images 115 may be used.

[0067] In step S403, the CPU 231 acquires a portion of the training data read in step S401, i.e., a mini-batch. This embodiment is an example in which the mini-batch method is used as a neural network training method. After acquiring a predetermined number of training data (mini-batch size, for example, 8), the CPU 231 proceeds to step S404. Note that the training method may be a method other than the mini-batch method, and for example, a batch method or an online method may also be used.

[0068] In step S404, the CPU 231 calculates the error in the neural network training. The error indicates the difference between the inference result of the model at that time and the correct answer information (correct answer label image). First, the training document image is input to the model to obtain the inference result. The inference result is, for example, a binary image in which pixels belonging to the boundaries of cells included in the training document image are set to 255 (255 if 8 bits / pixel) and other pixels are set to 0. The difference between the obtained inference result and the correct answer label image is evaluated pixel by pixel to obtain the error. The error is expressed, for example, by the following equation (1) (also called the loss function or error function). Note that equation (1) for calculating the error is just an example, and any other equation may be used as long as it is used for training the neural network.

[0069]

number

[0070] In formula (1), x k is a value indicating a pixel determined to belong to the boundary between cells in the inference result.k ' is a value indicating a pixel that is set as a pixel belonging to the boundary between cells in the correct labeled image. k y is a value indicating a pixel that is determined not to belong to the boundary between cells in the inference result. k ' is a value indicating pixels that are set as pixels that do not belong to cell boundaries in the correctly labeled image. For example, if a pixel belongs to a boundary, its value may be 255, and if it does not belong to a boundary, its value may be 0. M is the number of pixels that belong to cell boundaries in the correctly labeled image. N is the number of pixels that do not belong to cell boundaries in the correctly labeled image. That is, the first term of equation (1) is the sum of squares of the differences between pixels that belong to boundaries in the correctly labeled image and their corresponding pixels in the inference result. According to the first term of equation (1), the more pixels that are correctly inferred to belong to boundaries compared to pixels that do not belong to boundaries in the correctly labeled image, the smaller the loss. The second term is the sum of squares of the differences between pixels that do not belong to boundaries in the correctly labeled image and their corresponding pixels in the inference result. According to the second term of equation (1), the more pixels that are correctly inferred to not belong to boundaries compared to pixels that do not belong to boundaries in the correctly labeled image, the smaller the loss.

[0071] In this embodiment, the first and second terms do not need to be separated, and the loss function Loss=Σ(x k -x k ') 2 Alternatively, if the pixel values ​​of the correct labeled image and the inference result are given as binary values ​​of 0 and 1, the loss function can be set as Loss=Σ(x k +x k Here, the operation + is the exclusive OR, Σ is the sum of all pixels contained in the image, and x kと x k ' respectively represent corresponding pixels in the inferred image and the correct labeled image, and their values ​​may be, for example, 1 if they belong to the boundary and 0 if they do not.

[0072] In step S405, CPU 231 adjusts the model parameters. That is, the model parameters are changed by the backpropagation method based on the error calculated in step S404. As described above, in this example, the batch size is set to 8, so once the error is calculated using 8 pairs of learning document images and correct label data, the parameters are adjusted in S405.

[0073] In step S406, CPU 231 determines whether or not to end learning. This determination is made by judging whether or not the processes of steps S403 to S405 have been performed a predetermined number of times (for example, 1000 times). The predetermined number of times is determined by, for example, a user input at the start of this flowchart. If the predetermined number of times has been performed, the process proceeds to step S407. If not, the process proceeds to step S403, where model learning continues. This predetermined number of times may be a value obtained by dividing the number of prepared training data by the batch size. The learning model being learned, i.e., the cell boundary recognition model, is optimized by the loop of S403 to S406.

[0074] Finally, in step S407, the CPU 231 transmits the trained cell boundary recognition model as a trained model to the information processing server 103. The information processing server 103 stores the received cell boundary recognition model in the storage 265 within the information processing server 103.

[0075] <Recognition processing> Fig. 5 is an example of a flowchart illustrating the flow in which the information processing server 103 extracts item values ​​from a table in a document image in the sequence of Fig. 3(b). Note that an execution program for each step shown in Fig. 5 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 103, and is executed by the CPU 261 of the information processing server 103.

[0076] First, in step S501, the CPU 261 acquires the cell boundary recognition model stored in the storage 265 in step S305 in FIG. 3 and step S407 in FIG.

[0077] Next, in step S502, the CPU 261 acquires the document image 800 input by the user in S312 of Fig. 3. An example of the document image is shown in Fig. 8(a). The document image 800 includes a detailed table 801.

[0078] In step S503, CPU 261 inputs the document image into the cell boundary recognition model acquired in step S501 and performs inference processing, thereby outputting an inference result image 820 as shown in FIG. 8(b). Inference result image 820 includes inference results 821 that indicate the boundaries of the cells included in schedule 801. Inference result image 820 has a structure similar to that of the correct label image in FIG. 6A(b). That is, for each pixel, if it is recognized as a pixel that corresponds to a boundary between cells, a value indicating that it is a boundary (for example, 255, indicating a white part in the image) is stored, and if not, a value indicating that it is not a boundary (for example, 0, indicating a black part in the image) is stored.

[0079] In step S504, CPU 261 identifies each cell area based on the recognition result image acquired in step S503 and extracts item values. Here, a method for extracting item values ​​will be described with reference to FIG. 9. Assuming that inference result 821 for schedule 801 has been obtained as shown in FIG. 9, an example of extracting item values ​​from schedule 801 will be described. First, each cell area is identified from inference result 821 for schedule 801 obtained in step S503. Specifically, a closed area surrounded by a value indicating a cell boundary (a white portion in the image) is identified within inference result 821. By identifying the closed area, each cell area, such as cell areas 921, 922, 923, and 924, can be identified from inference result 821. Next, cells belonging to the same row as the identified cell areas are identified. To do this, first, cell area 921 extracted at the top left of document image 801 is selected from the extracted cell areas. Next, cells belonging to the same row as selected cell area 921 are extracted. For example, the coordinates of the upper left corner, width, and height of cell area 921 are (x, y, w, h). Here, x represents the horizontal position of the specification 801 in FIG. 9, y represents the position in the perpendicular direction, width w represents the length of the cell in the x direction, and height h represents the length of the cell in the y direction. In this way, cells other than cell area 921 whose y component of the upper left corner coordinate falls within the range of y±H to y+h±H (H is a predetermined constant) are identified. This makes it possible to identify cells belonging to the same row as the initially selected cell, such as cell areas 921, 922, 923, and 924. For other cell areas, similar processing can be used to identify each row in specification 801 and the cell areas contained therein. By interchanging x and y, and h and w in this explanation, it is also possible to identify cells belonging to the same column as the initially selected cell, and this may be done.

[0080] Next, character strings are extracted from cells for each row in statement 801. Character recognition processing is performed on each identified cell area to identify the character string. Character recognition can be performed using existing character recognition technology that identifies characters based on pixel information contained in the area. For example, character string 901, "product name," is extracted as the character string from cell area 921. Similarly, character strings 902, 903, and 904, "unit price," "quantity," and "amount," are extracted as the character strings from cell areas 922, 923, and 924. By performing the same processing for each row, the character strings in the cells for each row can be extracted as item values.

[0081] In step S505, the CPU 261 presents the item values ​​extracted in step S504 to the user. Fig. 10 shows an example of an item value confirmation screen. The confirmation screen 1000 is composed of a preview image display screen 1001, a result display screen 1002, and an end button 1013. The result display screen 1002 displays character strings 1005, 1006, 1007, and 1008 extracted from the cells. For example, the character string included in row 1004 of the schedule 1003 displayed on the preview image display screen 1001 is displayed. In addition, by pressing edit buttons 1009, 1010, 1011, and 1012, the results of the item values ​​1005, 1006, 1007, and 1008 can be corrected if they are incorrect. The preview image display screen 1001 highlights the schedule 1003 in the document image and row 1004 of the schedule output on the result display screen. The same applies to other lines; for example, by scrolling the result display screen 1002, the recognition results for other lines are displayed. When the end button 1013 is pressed, all processing ends. At this time, if the character string has been corrected by an editing operation, the original character string may be overwritten with the corrected character string and saved.

[0082] As described above, by targeting the boundaries between cells rather than the cell regions as the recognition target, it is possible to easily and efficiently create a correct labeled image. Then, by using a model trained using training data including the correct labeled image, it is possible to extract item values ​​in the table.

[0083] As mentioned above, the formula (1) for calculating the error is only an example, and the formula is not limited to the formula for calculating the sum of squares such as formula (1). For example, other functions for calculating the error, such as cross-entropy error or absolute error, may be used.

[0084] [Embodiment 2] In the first embodiment, a cell boundary recognition model was trained by creating correct labels to recognize cell boundaries. Generally, among all pixels in a document image, the number of pixels not belonging to cell boundaries is significantly greater than the number of pixels belonging to the boundaries. Therefore, when calculating loss using Equation (1), the loss attributable to pixels not belonging to cell boundaries tends to account for a larger proportion of the total loss (the second term in Equation (1) tends to be larger than the first term). Meanwhile, during training, the model attempts to minimize the total loss, but does not distinguish between pixels belonging to cell boundaries and pixels not belonging to cell boundaries (there is no distinction between the first and second terms in Equation (1)). Therefore, the training process tends to be biased toward pixels not belonging to cell boundaries, which can result in undesired learning results. For example, the model may fail to recognize cell boundaries.

[0085] In calculating the loss in embodiment 2, a class weight is introduced to calculate the overall loss by weighting the loss of the recognition result for pixels belonging to the boundary between cells. Note that, except for the loss function, the configuration of the system and device and other processing in this embodiment are the same as in embodiment 1, so a description of them will be omitted.

[0086] Fig. 12 is a flowchart illustrating the flow of learning a cell boundary recognition model by the learning device 102 in the second embodiment. Note that the execution program for each step shown in Fig. 12 is stored in either the ROM 232, RAM 234, or storage 235 of the learning device 102, and is executed by either the CPU 231 or GPU 239 of the learning device 102. Only the differences from the first embodiment will be explained. The difference from the first embodiment is that S404 in Fig. 4 is replaced with S1204 in Fig. 12.

[0087] In step S1204, the CPU 231 applies the class weights to calculate the error of the model. First, the learning document images of each learning data are input to the model to obtain an inference result. The difference between the obtained inference result and the correct label is evaluated to obtain the error. The error is expressed, for example, by the following formula (2). Note that formula (2) for calculating the error is just an example, and any formula that can be used for training a neural network may be used.

[0088]

number

[0089] In equation (2), x k and x k ',y k and y k ', M, and N are the same as in equation (1). The meaning of Σ is also the same. W is a value representing the weight in the class weight (the weight for the second term is 1, so it is omitted). The first term in equation (2) is the sum of squares of the differences between pixels that belong to the boundary in the correct labeled image and the corresponding pixels in the inference result, weighted by the class weight W. According to the first term in equation (2), the more pixels that are correctly inferred to belong to the boundary compared to pixels that belong to the boundary in the correct labeled image, the smaller the loss. The second term is the sum of squares of the differences between pixels that do not belong to the boundary in the correct labeled image and the corresponding pixels in the inference result. According to the second term in equation (2), the more pixels that are correctly inferred to not belong to the boundary compared to pixels that do not belong to the boundary in the correct labeled image, the smaller the loss.

[0090] As mentioned above, the second term, which represents the loss in the recognition results for pixels that do not belong to the boundary, generally tends to be larger, so by assigning a weight W to the first term, it is possible to adjust the balance between the magnitude of loss in the first and second terms. For this reason, it is desirable for the weight W to be a value greater than 1. Also, instead of multiplying the first term by the class weight W, it is also possible to use an equation in which the second term is divided by the weight W.

[0091] As described above, by introducing class weights in the second embodiment, model training can be made to tend to reduce unextracted boundaries between cells (making it easier to detect boundaries). By weighting the loss of recognition results for pixels that belong to boundaries heavier than the loss of recognition results for pixels that do not belong to boundaries, training proceeds to reduce the loss of recognition results for pixels that belong to boundaries. As a result, by using a trained model that uses the loss function of Equation (2), it becomes easier to detect boundaries between table cells.

[0092] When class weights are introduced into conventional cell area recognition methods, learning progresses so that fewer pixels belonging to cell areas are left unextracted, but there are cases where cell areas tend to be overdetected. In cell area recognition methods, the multiple cell areas being recognized are located close to each other, so the impact of overdetection is significant. In other words, cell areas can merge, making it difficult to recognize them as individual cells. On the other hand, in boundary recognition methods, class weights can also make it easier to detect boundaries (the boundaries tend to be detected as thicker), but because the boundaries are simply thicker, each cell area within the table can still be identified. Therefore, the introduction of class weights is more effective than conventional methods.

[0093] [Variations] In the first embodiment, cell boundaries contained in a document image are detected using a cell boundary recognition model. Then, cells are identified from the detected cell boundaries, and character strings contained in each cell are extracted. Alternatively, as in the above embodiment, an image showing the cell boundaries contained in a document image may be created, and then the image may be processed to create a correct label image showing the cell area. The created correct label image may then be used as training data to create a cell recognition model. For example, the correct label image may be created by inverting the values ​​of 1 (or 255) and 0 for pixels that form the interior of a table in the image showing the cell boundaries. This results in a correct label image showing the cell area. When training using this correct label image, the document image is input to the model during training, resulting in an image that identifies the cell portion, e.g., an image in which the inside of the cell is set to 1 (or 255) and the outside is set to 0 (this is called a cell-specific image). Therefore, the loss function may be, for example, the sum of squares of the differences between the correct label image showing the cell area and the cell-specific image.

[0094] The trained model created in this way can directly detect cells contained in a document image, and then extract characters from the document image in the area corresponding to the identified cells.

[0095] In the second embodiment, the loss function is evaluated using different terms for the boundary area and the area outside the boundary, and a class weight is applied to the boundary area. In the above modification, instead of the boundary area, the cell area and the area outside the cell area may be evaluated using different terms, and a class weight may be applied.

[0096] [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] Summary of embodiments The above embodiments can be summarized as follows: (Item 1) An image processing system using a trained model that receives a document image including a table as input and outputs the boundaries of the table cells included in the input document image, an acquisition means for inputting a document image into the trained model and acquiring boundaries of table cells included in the document image; a first specifying means for specifying a table cell area from the boundaries of the table cells; An image processing system comprising: (Item 2) Item 1, the image processing system a second acquiring means for acquiring an item value included in the cell from the cell area; and an output unit for outputting the item value. (Item 3) Item 2. The image processing system according to item 2, The output means displays a document image and the item value obtained based on the document image. (Item 4) Item 3. The image processing system according to item 3, The apparatus further includes an editing unit for editing the item value displayed by the output unit. An image processing system comprising: (Item 5) 5. The image processing system according to any one of items 1 to 4, The apparatus further includes an editing unit for editing the item value displayed by the output unit. An image processing system comprising: (Item 6) Item 5. The image processing system according to item 5, the learning means learns the trained model by machine learning using a document image including a table as an input and using cell boundaries of the table included in the input document image as an output; a means for acquiring training data in which a training document image is associated with a training image in which boundaries of table cells included in the training document image are identified; a means for inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; means for obtaining a value of a loss function using the boundary of the cell output by the learning model; means for optimizing the learning model so that the value of the loss function becomes small; The loss function is a function based on the error between the pixel at the boundary of the cell output by the learning model using the learning document image as an input and the pixel of the teacher image. An image processing system comprising: (Item 7) Item 6. The image processing system according to item 6, The loss function is a function that calculates the error between the pixels that belong to the boundary of a table cell included in the teacher image and the pixels that do not belong to the boundary of a table cell output by the learning model during learning using the learning document image as an input, for each of the pixels that belong to the boundary of a table cell and the pixels that do not belong to the boundary of a table cell included in the teacher image. An image processing system comprising: (Item 8) Item 7. The image processing system according to item 7, The loss function is a function that calculates the sum of a weighted value of the error between pixels that belong to the boundaries of table cells included in the teacher image and pixels that belong to the boundaries of table cells output by the learning model during training using the learning document image as an input, and an error between pixels that do not belong to the boundaries of table cells included in the teacher image and pixels that do not belong to the boundaries of table cells output by the learning model during training using the learning document image as an input. An image processing system comprising: (Item 9) The image processing system according to any one of items 6 to 8, The training image is acquired by a user specifying an area including the boundaries of table cells included in the learning document image. An image processing system comprising: (Item 10) 10. A program for causing a computer to function as the image processing system according to any one of items 1 to 9. (Item 11) An image processing method using an image processing device that uses a trained model through machine learning, in which a document image including a table is input and cell boundaries of the table included in the input document image are output, the image processing device comprising: an acquisition means; and a first identification means; the acquisition means inputs a document image into the trained model to acquire boundaries of table cells included in the document image; The first identifying means identifies a table cell area from the boundaries of the table cells. An image processing method comprising: (Item 12) A learning method for a trained model by machine learning, in which a document image including a table is input and the boundaries of the table cells included in the input document image are output, The computer acquiring training data in which a training document image and a training image in which boundaries of table cells included in the training document image are identified are associated with each other; a step of inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; Obtaining a loss function value using the cell boundaries output by the learning model; optimizing the learning model so that the value of the loss function becomes small; The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. A method for learning a trained model using machine learning. (Item 13) Item 13. A method for learning a trained model by machine learning according to Item 12, The loss function is a function that calculates the error between the pixels that belong to the boundary of a table cell included in the teacher image and the pixels that do not belong to the boundary of a table cell output by the learning model during learning using the learning document image as an input, for each of the pixels that belong to the boundary of a table cell and the pixels that do not belong to the boundary of a table cell included in the teacher image. A method for learning a trained model using machine learning. (Item 14) Item 14. The method for learning a trained model by machine learning according to Item 13, The loss function is a function that calculates the sum of a weighted value of the error between pixels that belong to the boundaries of table cells included in the teacher image and pixels that belong to the boundaries of table cells output by the learning model during training using the learning document image as an input, and an error between pixels that do not belong to the boundaries of table cells included in the teacher image and pixels that do not belong to the boundaries of table cells output by the learning model during training using the learning document image as an input. A method for learning a trained model using machine learning. (Item 15) A method for learning a trained model by machine learning according to any one of items 12 to 14, The training image is acquired by a user specifying an area including the boundaries of table cells included in the learning document image. A method for learning a trained model using machine learning. (Item 16) A machine learning trained model that takes a document image including a table as input and outputs the boundaries of the table cells included in the input document image, The computer acquiring training data in which a training document image and a training image in which boundaries of table cells included in the training document image are identified are associated with each other; a step of inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; Obtaining a loss function value using the cell boundaries output by the learning model; optimizing the learning model so that the value of the loss function becomes small; and The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. A trained model characterized by:

[0098] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0099] 100 Information processing system, 101 Information processing device, 102 Learning device, 103 Information processing server, 104 Network

Claims

1. An image processing system using a trained model that receives a document image including a table as input and outputs the boundaries of the table cells included in the input document image, an acquisition means for inputting a document image into the trained model and acquiring boundaries of table cells included in the document image; a first specifying means for specifying a table cell area from the boundaries of the table cells; An image processing system comprising:

2. 2. The image processing system according to claim 1, a second acquiring means for acquiring an item value included in the cell from the cell area; and an output unit for outputting the item value.

3. 3. The image processing system according to claim 2, The output means displays a document image and the item value obtained based on the document image. An image processing system comprising:

4. 4. The image processing system according to claim 3, The apparatus further includes an editing unit for editing the item value displayed by the output unit. An image processing system comprising:

5. 2. The image processing system according to claim 1, Further, the learning means performs machine learning to generate the trained model. An image processing system comprising:

6. 6. The image processing system according to claim 5, the learning means learns the trained model by machine learning using a document image including a table as an input and using cell boundaries of the table included in the input document image as an output; a means for acquiring training data in which a training document image is associated with a training image in which boundaries of table cells included in the training document image are identified; a means for inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; means for obtaining a value of a loss function using the boundary of the cell output by the learning model; means for optimizing the learning model so that the value of the loss function becomes small; The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. An image processing system comprising:

7. 7. The image processing system according to claim 6, The loss function is a function that calculates the error between the pixels that belong to the boundary of a table cell included in the teacher image and the pixels that do not belong to the boundary of a table cell output by the learning model during learning using the learning document image as an input, and the pixels that do not belong to the boundary of a table cell included in the teacher image. An image processing system comprising:

8. 8. The image processing system according to claim 7, The loss function is a function that calculates the sum of a weighted value of the error between pixels belonging to the boundaries of table cells included in the teacher image and pixels belonging to the boundaries of table cells output by the learning model during training using the learning document image as an input, and an error between pixels not belonging to the boundaries of table cells included in the teacher image and pixels not belonging to the boundaries of table cells output by the learning model during training using the learning document image as an input. An image processing system comprising:

9. 7. The image processing system according to claim 6, The training image is acquired by a user specifying an area including the boundaries of table cells included in the learning document image. An image processing system comprising:

10. A program for causing a computer to function as the image processing system according to any one of claims 1 to 9.

11. An image processing method using an image processing device that uses a trained model through machine learning, in which a document image including a table is input and cell boundaries of the table included in the input document image are output, the image processing device comprising: an acquisition means; and a first identification means; the acquisition means inputs a document image into the trained model to acquire boundaries of table cells included in the document image; The first identifying means identifies a table cell area from the boundaries of the table cells. An image processing method comprising:

12. A learning method for a trained model by machine learning, in which a document image including a table is input and the boundaries of the table cells included in the input document image are output, The computer acquiring training data in which a training document image is associated with a training image in which boundaries of table cells included in the training document image are identified; a step of inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; Obtaining a loss function value using the cell boundaries output by the learning model; optimizing the learning model so that the value of the loss function becomes small; The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. A method for learning a trained model using machine learning.

13. The method for learning a trained model by machine learning according to claim 12, The loss function is a function that calculates the error between the pixels that belong to the boundary of a table cell included in the teacher image and the pixels that do not belong to the boundary of a table cell output by the learning model during learning using the learning document image as an input, and the pixels that do not belong to the boundary of a table cell included in the teacher image. A method for learning a trained model using machine learning.

14. The method for learning a trained model by machine learning according to claim 13, The loss function is a function that calculates the sum of a weighted value of the error between pixels belonging to the boundaries of table cells included in the teacher image and pixels belonging to the boundaries of table cells output by the learning model during training using the learning document image as an input, and an error between pixels not belonging to the boundaries of table cells included in the teacher image and pixels not belonging to the boundaries of table cells output by the learning model during training using the learning document image as an input. A method for learning a trained model using machine learning.

15. The method for learning a trained model by machine learning according to claim 12, The training image is acquired by a user specifying an area including the boundaries of table cells included in the learning document image. A method for learning a trained model using machine learning.

16. A machine learning trained model that takes a document image including a table as input and outputs the boundaries of the table cells included in the input document image, The computer acquiring training data in which a training document image is associated with a training image in which boundaries of table cells included in the training document image are identified; a step of inputting the learning document image included in the training data and outputting table cell boundaries included in the learning document image using a learning model under training; Obtaining a loss function value using the cell boundaries output by the learning model; optimizing the learning model so that the value of the loss function becomes small; and The loss function is a function based on the error between the pixel at the boundary of the cell output from the learning model for the learning document image and the pixel of the teacher image. A trained model characterized by:

Images